Compare commits

...
Author SHA1 Message Date
John Sully 240d6886ff release 0.9.5 2019-05-09 20:14:19 -04:00
John Sully 272d9297b6 Fix CentOS build failure 2019-05-09 19:54:34 -04:00
John Sully f5bfc71899 Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-05-09 19:16:50 -04:00
John Sully 42e1ded55f build break fix 2019-05-09 19:16:31 -04:00
John Sully 6c410af5a8 Fix issue #11 2019-05-09 19:14:10 -04:00
John Sully 57ae676181 Fix typo 2019-05-09 19:06:58 -04:00
John Sully 7896ca6e52 Add missing file 2019-05-09 19:05:09 -04:00
John Sully c4a315dd5e Change debug messages to reference KeyDB 2019-05-09 19:03:50 -04:00
John Sully bf26959b72 Fix module locking issues 2019-05-09 19:00:27 -04:00
John Sully 37b2d046be refactor server global into a pointer (so that threadsafe commands can set this to NULL to prove they don't rely on it 2019-04-21 14:01:10 -04:00
John Sully fa2e407873 Factor out some normally const globals 2019-04-21 13:20:45 -04:00
John Sully c61cf57db5 additional fix to rreplay forward compat 2019-04-20 01:36:23 -04:00
John Sully ef3690f2db Make rreplay forward compatible with extra future args 2019-04-20 01:15:01 -04:00
John Sully 14d01ec7bd Tweak mvcc time format so rollovers are not an issue 2019-04-20 01:03:08 -04:00
John Sully 5dc9f70386 Timestamp writes to objects so we can be smarter about merging databases 2019-04-20 00:52:07 -04:00
John Sully 88e593bf96 Fix use uninitialized 2019-04-19 23:41:40 -04:00
John Sully efde2e6be6 Additional MVCC work and fix memory leak loading objects from rdb 2019-04-19 22:54:42 -04:00
John Sully 5a78a3bc46 Add version override command for users that need compatibility with Redis 2019-04-19 19:57:11 -04:00
John Sully b276d803d4 RREPLAY PERF: The UUID stack is unnecessary, RREPLAY itself can detect the cycle and call cancel() 2019-04-19 19:36:47 -04:00
John Sully da51ac430b Detect cycles in mesh networks with multi master 2019-04-19 19:31:11 -04:00
John Sully 01eb0a3c7f Fix crash near boot when adding replicas and dumping database 2019-04-19 17:08:54 -04:00
John Sully 715312872d Bump version due to very bad bug 2019-04-18 22:53:03 -04:00
John Sully 2b4efdaaa8 Replication fake connection should inhereit the user and permission of the real connection 2019-04-18 22:52:41 -04:00
John Sully dc2b3cf4f8 Replication fake connection should inhereit the user and permission of the real connection 2019-04-18 22:07:35 -04:00
John Sully c4621a5ed2 Start of MVCC support (and more C++) 2019-04-16 23:16:03 -04:00
John Sully 07f5596355 Start of MVCC 2019-04-16 23:13:14 -04:00
John Sully b4ad365758 bump version 2019-04-16 13:35:35 -04:00
John Sully 3aedcbc0af Fix module linking and crashing issues 2019-04-16 13:34:56 -04:00
John Sully 1c953a4d5b Fix module linking and crashing issues 2019-04-16 13:34:22 -04:00
John Sully 85e62208b3 Fix replica authentication failure when masterauth is below replicaof in the config file 2019-04-15 22:31:36 -04:00
John Sully c64ce7aa9c Fix replica authentication failure when masterauth is below replicaof in the config file 2019-04-15 22:31:02 -04:00
John Sully acdc4fdfec Fix memory leak unsetting master 2019-04-11 19:55:44 -04:00
John Sully e951386265 Fix memory leak unsetting master 2019-04-11 19:55:08 -04:00
John Sully 715de15c3a Merge branch 'unstable' into RELEASE_0_9 2019-04-11 19:43:50 -04:00
John Sully 7ef9c18b72 Fix compile errors when MALLOC=memkind 2019-04-11 19:40:45 -04:00
John Sully 309dcdfd55 Bump version 2019-04-11 19:25:50 -04:00
John Sully cac378a1e6 Merge branch 'unstable' into RELEASE_0_9 2019-04-11 19:20:53 -04:00
John Sully a69b4e79db Fix failure to respect masterauth issue #26 2019-04-11 19:08:12 -04:00
John Sully 740fc6c2b8 Fix addition RESP3 regressions (return empty set) 2019-04-08 16:52:57 -04:00
John Sully 42202c868d Fix protocol break with hgetall and family 2019-04-08 13:52:02 -04:00
John Sully 8c133b605c Move remaning files dependent on server.h over to C++ 2019-04-08 01:00:48 -04:00
John Sully b50acd7238 Move dict to C++ 2019-04-07 17:23:59 -04:00
John Sully 54db278695 Port db to C++ 2019-04-07 17:13:14 -04:00
John Sully 011948bcef Move a bunch of files to C++ 2019-04-07 16:54:56 -04:00
John Sully 9f95d8b38f Make t_string C++ 2019-04-07 15:24:23 -04:00
John Sully 3d394a6fa2 Fix clang compile error 2019-04-06 00:20:17 -04:00
John Sully 68d2b648e0 Add RREPLAY command and put the issue of packet storms to rest. Fixes issue #24 2019-04-06 00:14:27 -04:00
John Sully 283998404f Fix regression in active replication caused by multimaster PR 2019-04-04 16:50:40 -04:00
John Sully a7b4d0fea6 Merge pull request #23 from JohnSully/wip-multimaster
Wip multimaster
2019-04-04 11:40:58 -07:00
John Sully 59051dd59d All tests runs pass! YAYgit add src/cluster.c src/config.c ! 2019-04-04 01:02:19 -04:00
John Sully cd836a37e7 Fix some more failing tests 2019-04-04 00:25:30 -04:00
John Sully 4326d22380 Merge branch 'unstable' into wip-multimaster 2019-04-03 00:12:13 -04:00
John Sully 161d3a35ff Fix cluster test failure (not real issue) 2019-04-03 00:11:34 -04:00
John Sully 20a9d21dfb Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-04-03 00:09:51 -04:00
John Sully e902789aa7 Fix failing tests due to build issue inhereted from Redis 2019-04-03 00:09:26 -04:00
John Sully 3745bbd95b Merge branch 'unstable' of https://github.com/antirez/redis into unstable 2019-04-03 00:00:38 -04:00
John Sully e63b2880b0 Cluster tests shouldn't crash (note: still not passing either) 2019-04-02 22:32:10 -04:00
John Sully a9b70e1ef7 Multiple masters is no good if we keep dropping the database 2019-04-02 21:47:21 -04:00
John Sully e300f753c5 Tests are passing! YAY! 2019-04-02 21:36:04 -04:00
John Sully efaeca5887 It compiles and doesn't crash immediately! 2019-04-02 16:47:05 -04:00
John Sully 4d67305661 Link to GUI 2019-03-31 14:37:54 -04:00
John Sully 853bb6ca3e Commit missing file 2019-03-28 17:56:14 -04:00
John Sully b16f23ae8e Travis needs to install uuid-dev 2019-03-28 16:18:26 -04:00
John Sully 91e6368de0 Implement database merging for Active Replicas 2019-03-28 15:12:43 -04:00
John Sully e4451b9628 Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-03-27 22:59:38 -04:00
John Sully 770bb8ef5f If no rdb file is specified and no s3 file is specified use the default RDB filename. If only S3 is specified use diskless sync 2019-03-27 22:59:12 -04:00
John Sully 87b958a210 Update README.md 2019-03-27 21:52:08 -04:00
John Sully 314ac90ba9 Update README.md 2019-03-27 15:39:25 -04:00
John Sully 9f33822083 Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-03-26 20:39:56 -04:00
John Sully 10a4b35bc5 SIGALRM should happen on a worker thread to make our tests happy 2019-03-26 20:39:30 -04:00
John Sully 7b1dd0b60d Make PSYNC2 tests more reliable on slower hardware 2019-03-26 18:59:31 -04:00
John Sully 7a650741f7 Merge pull request #18 from DevopsDood/unstable
Update Readme for missing docker flag
2019-03-26 09:49:53 -04:00
Richard Silver 2ce5f1b944 Update Readme for missing docker flag 2019-03-25 15:55:23 -07:00
John Sully 063a14372d Give information on dependencies that need installation. 2019-03-25 16:22:10 -04:00
John Sully dd70e1b571 Add depenednecy on uuid-dev to dockerfile 2019-03-25 02:09:53 -04:00
John Sully d7112df465 Bump version because we added new features 2019-03-25 01:12:31 -04:00
John Sully ed021c88b9 Merge branch 'unstable' into RELEASE_0_9 2019-03-25 01:11:50 -04:00
John Sully 1840b893c2 Talk about active replica 2019-03-25 00:12:04 -04:00
John Sully 8bcde757ce We don't corretly terminate our varargs 2019-03-24 18:57:25 -04:00
John Sully 1dff223c71 OS X build fixes 2019-03-24 18:06:57 -04:00
John Sully a7aa2b0740 Active Replica Support 2019-03-24 15:39:10 -04:00
John Sully fb45fefbca Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-03-21 23:11:55 -04:00
John Sully 7b914c2531 Fix kqueue build error 2019-03-21 23:11:19 -04:00
John Sully 75d517a27f Merge pull request #16 from gitter-badger/gitter-badge
Add a Gitter chat badge to README.md
2019-03-21 21:57:25 -04:00
John Sully 4c98bc690e Update README.md 2019-03-21 21:26:00 -04:00
John Sully 550b3a3a99 Update README.md 2019-03-21 21:25:49 -04:00
The Gitter Badger ef7e89352d Add Gitter badge 2019-03-22 01:24:32 +00:00
John Sully 9322d604ee Merge branch 'unstable' of https://github.com/antirez/redis into unstable 2019-03-21 20:15:59 -04:00
John Sully 5d92110630 Merge pull request #15 from DevopsDood/unstable
Added docker builder for those that want it
2019-03-21 19:54:00 -04:00
Richard Silver 4056037cc2 Merge branch 'unstable' of github.com:JohnSully/KeyDB into unstable 2019-03-21 16:19:45 -07:00
Richard Silver 677ae5487f updated readme and Dockerfile to allow for tests to be run in container 2019-03-21 16:18:37 -07:00
John Sully 19956b3807 comment issue 2019-03-21 22:18:48 +00:00
John Sully ab9122f2cb Polarity of the weak flag was wrong. We want to use the strong variant of atomic_compare_exchange always 2019-03-21 22:17:04 +00:00
Richard Silver e62c6b0ed0 Added build Dockerfile 2019-03-21 15:15:11 -07:00
John Sully 27a420fbc2 Compile issues at O0 optimization 2019-03-21 21:57:18 +00:00
John Sully b775ebb382 ARM build fix: Don't use X64 asm here 2019-03-20 04:14:33 +00:00
John Sully ac80a5c6a6 Add debugging stats to the INFO command 2019-03-19 22:04:33 -04:00
John Sully c74100a552 Add release badge 2019-03-19 20:54:36 -04:00
John Sully 03379106d6 Add CI badge 2019-03-19 20:46:25 -04:00
John Sully 9ad229f663 Getting close! Maybe there was a better way to do this than checking in travis.yml files repeatedly but alas it works and doesn't affect people building locally 2019-03-19 20:40:17 -04:00
John Sully 0900edf776 Clang build with libc malloc 2019-03-19 20:34:09 -04:00
John Sully ea520027b9 more clang travis fixes 2019-03-19 20:26:35 -04:00
John Sully 04679b3b3b Fix race condition building dependencies with multithreaded builds 2019-03-19 20:25:30 -04:00
John Sully 2bf5c1324a Will clang build now with travis? 2019-03-19 20:00:52 -04:00
John Sully 3d93ca1bf6 clang build failure 2019-03-19 19:55:03 -04:00
John Sully c548431611 Travis clang dependencies 2019-03-19 19:50:28 -04:00
John Sully 0ab29f7394 Makefile ignores env CXXFLAGS 2019-03-19 19:29:14 -04:00
John Sully 30dab7ad14 No underscore 2019-03-19 19:22:27 -04:00
John Sully 0c5fa59221 Maybe this will make travis-ci stop using GCC headers when compiling with clang 2019-03-19 19:12:15 -04:00
John Sully 267e84f97d Feature gate SO_INCOMING_CPU so we compile on older kernels 2019-03-19 18:55:42 -04:00
John Sully 344cca75c6 typo 2019-03-19 18:40:50 -04:00
John Sully 0dd8e682f4 It would be nice if our CI actually called make... 2019-03-19 18:37:53 -04:00
John Sully da7e54819a Fix issues with travis config 2019-03-19 18:31:45 -04:00
John Sully 4e70215307 Let's try travis-ci 2019-03-19 18:27:54 -04:00
John 6aa162f6ff Fix linker errors on some compilers due to sdslen() 2019-03-19 22:08:20 +00:00
John Sully 5d360ed401 Bump the version to indicate this updated release has bug fixes. 2019-03-19 15:38:39 -04:00
John Sully 8164c93d5a Merge branch 'unstable' into RELEASE_0_9 2019-03-19 15:24:42 -04:00
John Sully d380074569 KeyDB has different aims than the Redis project.
We believe software is for the user.  If a feature requires complicating the codebase
but simplifies our user's lives we will do it any time.  This should not be taken
as an outright rejection of the manifesto, as we agree on many points.
2019-03-19 01:36:03 -04:00
John Sully c406e7ff51 Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-03-19 01:26:22 -04:00
John Sully 31ff6cffee Merge branch 'unstable' of https://github.com/antirez/redis into unstable 2019-03-19 01:26:10 -04:00
John Sully a9e525c81e Hack to prevent build errors on some machines due to missing throw() 2019-03-15 05:28:20 +00:00
John Sully 6157a5fc1b Merge /home/john/repos/redis_clean into unstable 2019-03-14 14:22:41 -04:00
John Sully be3cb1ad33 Merge branch 'unstable' of https://github.com/antirez/redis into unstable 2019-03-14 14:22:27 -04:00
John Sully a89ab4d6b0 Fix hyperloglog corruption 2019-03-14 14:02:16 -04:00
John Sully 966f4a8144 Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-03-13 18:08:59 -04:00
John Sully 261cf24efc Merge branch 'unstable' of https://github.com/antirez/redis into unstable
Lots of fixes and improvements from upstream.
2019-03-13 18:08:22 -04:00
JohnSully a3da10f4eb Update README.md 2019-03-13 16:57:40 -04:00
John Sully 6cb8063497 Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-03-13 16:53:55 -04:00
John Sully a45f212693 Implement load database dumps from S3. We already save. 2019-03-13 16:53:37 -04:00
JohnSully 7c72cd5471 Merge pull request #9 from 0xflotus/patch-1
fixed guarantees
2019-03-13 12:22:29 -04:00
0xflotus 66f2032708 fixed guarantees 2019-03-13 15:14:34 +01:00
JohnSully 7546c7d8b4 Update README.md 2019-03-12 14:38:03 -04:00
JohnSully 82621702cf Merge pull request #6 from JohnSully/unstable
Merge readme changes
2019-03-12 01:30:56 -04:00
JohnSully 703e4ad2fe formatting fixes 2019-03-12 00:59:28 -04:00
JohnSully 0b98280a1f Add charts to readme
Hotlink from medium, hope they don't mind.
2019-03-12 00:55:03 -04:00
John Sully 4609bed6a4 Merge branch 'unstable' of https://github.com/JohnSully/redis2 into RELEASE_0_9 2019-03-11 20:54:14 -04:00
John Sully 99ccb11d89 Werror is good for me, but not for people compiling on other compilers 2019-03-11 20:53:57 -04:00
JohnSully 8a5a513662 Explain how multithreading works 2019-03-11 20:37:57 -04:00
JohnSully bf0fd22309 Update README.md 2019-03-11 19:38:20 -04:00
John Sully cc98d10fc6 Make storage.h C++ safe 2019-03-09 13:42:08 -05:00
John Sully a6a022faa2 sizeof is too big 2019-03-09 13:41:49 -05:00
John Sully a864ad1c9c bump version 2019-03-07 19:13:59 -05:00
John Sully 0bc586933f Merge branch 'unstable' into RELEASE_0_9 2019-03-07 19:13:33 -05:00
John Sully 2785a8b4d4 Module threading fixes 2019-03-07 19:13:01 -05:00
John Sully d5d5f6ae06 Merge branch 'unstable' into RELEASE_0_9 2019-03-06 16:39:15 -05:00
John Sully be3ba1d53e Limit server-threads to cores in the machine 2019-03-06 16:39:02 -05:00
John Sully 09e6a4fee0 port server.c to server.cpp 2019-03-06 16:29:30 -05:00
John Sully efc60dd8c5 Set version 2019-03-06 15:25:28 -05:00
John Sully a52cd974b9 Don't send replies to slaves after we've queued them to be closed 2019-03-06 15:21:05 -05:00
John Sully de4217489c defer free 2019-03-05 15:46:19 -05:00
John Sully 3c658b5012 Fix crash with paused clients (accessed when global lock is not acquired) 2019-03-03 22:24:12 -05:00
John Sully f6814aa825 Crash in Memory Doctor when all clients are slaves 2019-03-03 13:56:47 -05:00
John Sully 01a529cbda Fix issue #2, check posix return values 2019-03-02 20:43:03 -05:00
John Sully a1475af1e4 sds copies uninitialized data wasting time for everyone 2019-03-02 19:27:52 -05:00
John Sully 07594d4f95 Merge branch 'unstable' of https://github.com/antirez/redis into unstable
We want that awesome multithreaded benchmark
2019-03-02 18:07:18 -05:00
John Sully bad73faf28 write fastlock_unlock in ASM... because its faster 2019-03-02 16:47:27 -05:00
John Sully 8d3704d93b Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-03-01 16:40:20 -05:00
John Sully 8bd56fadd6 tighten up the spinlock loop, and some other bikeshedding 2019-03-01 13:29:21 -05:00
John Sully 23f9b584a8 remove unnecessary logs 2019-02-28 13:57:58 -05:00
JohnSully 141557963e Update README.md 2019-02-26 15:16:46 -05:00
John Sully ffabf3c5b7 Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-02-26 15:14:50 -05:00
John Sully 9b1e95809c Thread affinity feature (default disabled) 2019-02-26 15:14:35 -05:00
JohnSully b8776bb001 Link to detailed benchmarking results and methodology. 2019-02-26 15:11:25 -05:00
JohnSully 84c20e8337 More fine tuning help. Hopefully we can automate this. 2019-02-26 03:30:37 -05:00
JohnSully bae2409cce Update benchmarking notes. 2019-02-26 02:30:03 -05:00
John Sully 8f7c44257b test code shouldn't be commited 2019-02-26 00:21:37 -05:00
John Sully 2772ce249c Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-02-26 00:20:11 -05:00
John Sully bcfd9327cb writes shouldn't be under the global lock 2019-02-26 00:19:38 -05:00
John Sully 01ff36fc28 Option to disable ASM spinlock 2019-02-26 00:00:04 -05:00
John Sully 6466b09fdc Fix build break 2019-02-26 01:10:07 +00:00
John Sully 40d7a701fe Rewrite our spinlock in assembly, pretty big performance improvement 2019-02-25 18:21:27 -05:00
JohnSully 78b9d303d3 Formatting (more) 2019-02-23 03:08:24 -05:00
John Sully d912013d1b Update copyright to include additional authorship 2019-02-23 02:27:19 -05:00
John Sully 898e6519d1 Merge branch 'unstable' of https://github.com/JohnSully/redis2 into unstable 2019-02-23 02:25:00 -05:00
John Sully 8c074a7d01 Merge branch 'Multithread' into unstable 2019-02-23 02:24:23 -05:00
John Sully 06d4b7efc3 Support configurable number of threads 2019-02-23 01:51:04 -05:00
John Sully 78451a4223 Support a non-spinlock mutex if compiled in 2019-02-23 01:50:02 -05:00
JohnSully 6e69b4a795 More edits 2019-02-23 01:48:05 -05:00
JohnSully 1c36b46cb4 type 2019-02-23 01:33:27 -05:00
JohnSully 73abd04b95 Formatting 2019-02-23 01:32:50 -05:00
JohnSully 29d6845d1f Initial rewrite of Readme 2019-02-23 01:28:44 -05:00
John Sully 3871725979 All tests pass single thread 2019-02-23 00:09:34 -05:00
John Sully 199269bff7 Fix failure to trim querybuf (and memory tests that fail because of it) 2019-02-22 22:09:25 -05:00
John Sully 0ebf0f7338 Fix EMBSTR corruption 2019-02-22 21:32:43 -05:00
John Sully 0ccceddae0 Missing file from merge 2019-02-22 21:16:50 -05:00
John Sully 03646fe492 Disable RTTI, we don't need it 2019-02-22 21:16:27 -05:00
John Sully 8b72fe935e Merge branch 'unstable' of https://github.com/antirez/redis into Multithread 2019-02-22 21:16:10 -05:00
John Sully 1761aabab4 Lock use after free 2019-02-22 21:00:14 -05:00
John Sully 934375cfea fix corruption in object size optimization 2019-02-22 17:45:50 -05:00
John Sully b6735ac883 Remove unnecessary mfences 2019-02-22 15:49:22 -05:00
John Sully f4b060e0bd Prevent mixed up client replies, and deadlocks 2019-02-22 01:24:16 -05:00
John Sully d016b967c5 Implement try_lock 2019-02-22 01:23:31 -05:00
John Sully ebf0ae3e97 Merge branch 'unstable' of https://github.com/antirez/redis into Multithread 2019-02-21 18:17:12 -05:00
John Sully 41164fbb61 Change default back to jemalloc (until defrag is implemented) and fix compile errors 2019-02-21 01:36:38 -05:00
John Sully 75f589c702 Don't commit core dumps 2019-02-21 00:43:09 -05:00
John Sully ac4c00d6d9 Rebrand more strings visible on console, we're not allowed to call ourselves Redis 2019-02-21 00:24:03 -05:00
John Sully 1f2f2edfac Update copyright notice on files heavily changed 2019-02-21 00:17:07 -05:00
John Sully 942510f1a5 Change fastlock implementation from naive spinlock to a ticket lock for better fairness and lower peak latency 2019-02-21 00:16:47 -05:00
John Sully 02e7fe400c Cleanup lock contention, and ensure clients are written to in an unsafe way when the global lock is released 2019-02-20 23:30:21 -05:00
John Sully 29c1105132 Multithreading works! 2019-02-20 01:20:26 -05:00
John Sully 357a9e92e2 Prevent erroneously setting CLIENT_PENDING_WRITE when we have async writes 2019-02-19 01:37:16 -05:00
John Sully bf41d3916e Most tests failing, except some memory and number of PSYNC syncs 2019-02-19 01:11:00 -05:00
John Sully acbad0c04e deadlock fixes 2019-02-18 23:52:21 -05:00
John Sully 2526d51d1a Thread safety fixes 2019-02-18 22:25:35 -05:00
John Sully a7bed6327a Extra AE functionality 2019-02-18 18:56:45 -05:00
John Sully a47b0f4d3b Load balance connections across threads (config NYI and hardcoded) 2019-02-16 14:26:19 -05:00
John Sully 2c220889c6 Fix recursive fastlock... 2019-02-16 14:25:14 -05:00
John Sully 5fc8747feb make headers C++ safe 2019-02-15 16:55:40 -05:00
John Sully 6d8530d493 mitigate race condition in test 2019-02-15 16:55:19 -05:00
John Sully 62090d0a97 make networking.c a C++ file 2019-02-15 14:11:34 -05:00
John Sully f5caec488d Make the lock recursive, this is because processEventsWhileBlocked may cause us to lock multiple times 2019-02-15 14:11:05 -05:00
John Sully 48f6d0d800 Fix most failing tests 2019-02-15 10:53:32 -05:00
John Sully d62178ec8c Initial work of multithreaded key-db. Note: Fails tests 2019-02-11 03:36:18 -05:00
John Sully da84b26785 Fastlock fixes 2019-02-10 22:00:19 -05:00
John Sully 988ea40101 Fastlock fixes 2019-02-10 22:00:19 -05:00
John Sully eb9070c833 move ae to C++ 2019-02-10 20:24:11 -05:00
John Sully c4dbc557eb move ae to C++ 2019-02-10 20:24:11 -05:00
John Sully b7b503d3ce Keep pool allocator up to date (even though we aren't enabling it) 2019-02-10 19:33:07 -05:00
John Sully 180c93ea4f Keep pool allocator up to date (even though we aren't enabling it) 2019-02-10 19:33:07 -05:00
John Sully 327bbec981 Prevent so many copies of sdslen getting added 2019-02-10 19:31:46 -05:00
John Sully 44299002e0 Prevent so many copies of sdslen getting added 2019-02-10 19:31:46 -05:00
John Sully 7175a8e4eb Performance enhancements! 2019-02-09 13:05:06 -05:00
John Sully edd87115f1 Performance enhancements! 2019-02-09 13:05:06 -05:00
John Sully 9aefc183c6 Reduce memory usage for in place strings by 8 bytes 2019-02-09 13:04:18 -05:00
John Sully 2f9d958e96 Reduce memory usage for in place strings by 8 bytes 2019-02-09 13:04:18 -05:00
John Sully 3e9b867709 complete rebranding with tests passing 2019-02-09 10:11:46 -05:00
John Sully d3fc46b3a8 complete rebranding with tests passing 2019-02-09 10:11:46 -05:00
John Sully e8b5bbf7a1 Fix bug where we try to load a database with no name 2019-02-09 07:56:45 -05:00
John Sully d8b67e717c Fix bug where we try to load a database with no name 2019-02-09 07:56:45 -05:00
John Sully 79ef18e885 remove Redis branding 2019-02-08 17:56:32 -05:00
John Sully e4bcae6156 remove Redis branding 2019-02-08 17:56:32 -05:00
John Sully 23a91df9f6 Support AWS S3 saving via the s3 cli tools 2019-02-06 01:06:48 -05:00
John Sully e38d1e6c7f Support AWS S3 saving via the s3 cli tools 2019-02-06 01:06:48 -05:00
John Sully d8741595ae Merge branch 'unstable' of https://github.com/antirez/redis into unstable 2019-02-06 00:09:39 -05:00
John Sully 3e9e84ca19 Merge branch 'unstable' of https://github.com/antirez/redis into unstable 2019-02-06 00:09:39 -05:00
John Sully 3c9dd6ffc2 Make main headers C++ safe, and change rdb to use file descriptor instead of FILE pointer 2019-02-05 23:36:40 -05:00
John Sully 41a24e689e Make main headers C++ safe, and change rdb to use file descriptor instead of FILE pointer 2019-02-05 23:36:40 -05:00
antirez 20ba1b7e96 ACL: redis.conf: mark old ACL-alike stuff as deprecated. 2019-02-05 17:59:05 +01:00
antirez 475fd7ba2e ACL: ability to configure an external ACL file. 2019-02-05 17:49:52 +01:00
antirez 15a76e7868 ACL: change behavior of redefined user. Last line counts. 2019-02-05 10:52:05 +01:00
antirez 8ce3c16317 ACL: implement rewriting of users in redis.conf. 2019-02-05 10:48:17 +01:00
John Sully 42d28c9ff6 complete malloc memory class work, and pass tests 2019-02-04 16:56:13 -05:00
John Sully 2f753a3539 complete malloc memory class work, and pass tests 2019-02-04 16:56:13 -05:00
antirez b74c41c8e4 ACL: fix user/rule inverted error message. 2019-02-04 16:58:35 +01:00
antirez 58c070757d ACL: load the defined users at server startup. 2019-02-04 16:39:07 +01:00
antirez 0082503bdf ACL: implement ACLLoadConfiguredUsers(). 2019-02-04 16:35:15 +01:00
antirez 20fa89d093 ACL: better error reporting in users configuration errors. 2019-02-04 13:04:35 +01:00
antirez 0d6e0f8d23 ACL: make ACLAppendUserForLoading() able to report bad argument. 2019-02-04 13:00:58 +01:00
antirez 1327538221 ACL: initial appending of users in user loading list. 2019-02-04 12:55:48 +01:00
John Sully 82372cbce8 Properly integrate memkind 2019-02-01 15:21:00 -05:00
John Sully 7d76a8d602 Properly integrate memkind 2019-02-01 15:21:00 -05:00
John Sully 69b07c5cc2 Merge commit '09a263e6076b3fd0fe3b094a2bfad7f93fc997d6' as 'deps/memkind/src' 2019-02-01 14:45:39 -05:00
John Sully d3a31390b5 Merge commit '09a263e6076b3fd0fe3b094a2bfad7f93fc997d6' as 'deps/memkind/src' 2019-02-01 14:45:39 -05:00
John Sully 09a263e607 Squashed 'deps/memkind/src/' content from commit bb9f19dd1
git-subtree-dir: deps/memkind/src
git-subtree-split: bb9f19dd1b3ed6cc5e1b35919564ccf6f4b32f69
2019-02-01 14:45:39 -05:00
John Sully a6d27dc02c template config 2019-02-01 14:01:35 -05:00
John Sully b290c49a40 template config 2019-02-01 14:01:35 -05:00
John Sully bbcaf48cb0 Support configurable file backed scratch 2019-02-01 13:54:59 -05:00
John Sully a476fdfd8c Support configurable file backed scratch 2019-02-01 13:54:59 -05:00
antirez a23e926924 ACL: implement ACLAppendUserForLoading(). 2019-02-01 13:02:59 +01:00
antirez 816f2fce08 ACL: skeleton and first ideas for postponed user loading. 2019-02-01 12:20:09 +01:00
antirez 38c6030278 ACL: set modules help clients to the root user.
It does not make much sense to limit what modules can do: the admin
should instead limit what module commnads an user may call. So
RedisModule_Call() and other module operations should be able to execute
everything they want: the limitation should be posed by the API exported
by the module itself.
2019-02-01 11:37:28 +01:00
antirez 15385d4d68 ACL: assign ACL command ID to modules commands. 2019-02-01 08:17:24 +01:00
antirez 91f55d3206 ACL: implement DELUSER. 2019-01-31 18:33:14 +01:00
antirez b807d63f92 ACL: check arity of LIST / USERS subcommand. 2019-01-31 18:32:49 +01:00
antirez 3ec2dc3a6d ACL: don't emit useless flags in ACLDescribeUser(). 2019-01-31 17:04:42 +01:00
antirez 1cb9b66aaa ACL: implement LIST and USERS subcommands. 2019-01-31 17:01:32 +01:00
antirez 110c8caf8a ACL: flags refactoring, function to describe user. 2019-01-31 16:49:22 +01:00
antirez 2458e62da4 ACL: add function to return ACLSetUser() string errors. 2019-01-30 16:02:25 +01:00
antirez 88af62aca2 ACL: don't allow patterns after the * pattern. 2019-01-30 15:59:45 +01:00
antirez e1d14fb943 ACL: implement keys field in ACL GETUSER. 2019-01-30 15:52:36 +01:00
antirez 7deb1e90d9 Acl: Test: check command rules synthesis. 2019-01-30 12:01:37 +01:00
antirez cd60bcc427 ACL: free memory leak when freeing subcommands array. 2019-01-30 11:50:30 +01:00
antirez 0fa16cbd4b ACL: return error when adding subcommands of fully added commands.
It's almost certainly an error from the user side.
2019-01-30 08:25:12 +01:00
antirez ef252265d9 ACL: remove leak in ACLSetUser(). 2019-01-30 08:20:31 +01:00
antirez 1f64563801 ACL: ACLDescribeUserCommandRules(): emit subcommands. 2019-01-30 08:17:33 +01:00
antirez 20106cfa04 ACL: clear the subcommands array when setting category bits. 2019-01-30 08:09:05 +01:00
John Sully 85194d99da crash fixes 2019-01-30 01:15:10 -05:00
John Sully 5e6b9b9c54 crash fixes 2019-01-30 01:15:10 -05:00
John Sully 5c86d2faa2 Custom flash heap 2019-01-29 18:10:46 -05:00
John Sully 0ffcf355fe Custom flash heap 2019-01-29 18:10:46 -05:00
antirez 3293d1c2e7 ACL: finish/fix ACLDescribeUserCommandRules() first version. 2019-01-29 18:54:21 +01:00
antirez 40e619cff2 ACL: ACLDescribeUserCommandRules(): add final commands. 2019-01-29 18:41:11 +01:00
antirez 66714ee449 ACL: initial design for ACLDescribeUserCommandRules() and helpers. 2019-01-29 17:25:05 +01:00
antirez 02f4e087d7 ACL: enforce ACLs in Lua scripts as well. 2019-01-29 10:12:22 +01:00
antirez 58404e2e12 ACL: refactoring and fix of adding subcommands to users. 2019-01-28 18:40:54 +01:00
antirez 95fe993018 ACL: Test: check subcommands (test fails). 2019-01-28 18:29:22 +01:00
antirez 25d1876405 ACL: reset the subcommands table on +@all / -@all.
This also is a bugfix because after -@all the previously enabled
subcommands would remain valid.
2019-01-28 18:27:42 +01:00
antirez 2fbd532d61 ACL: fix crash when checking for subcommands. 2019-01-28 18:15:59 +01:00
antirez 7b1493a249 ACL: Test: nopass user flag, commands/groups +/- rules. 2019-01-28 12:40:07 +01:00
antirez b03f0c0403 ACL: Test: check default behavior and keys ACLs. 2019-01-28 12:33:18 +01:00
antirez fa0ce5aee7 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-01-28 12:13:25 +01:00
antirez b87b3bb812 ACL: Test: add the acl tests as default unit. 2019-01-28 12:12:04 +01:00
antirez 8dc96916f1 ACL: remove leak when removing passwords/patterns. 2019-01-28 12:11:11 +01:00
antirez c2865f4810 ACL: Test: check passwords validity and removal. 2019-01-28 12:06:25 +01:00
antirez 619017f9c2 ACL: Test: enabled/disabled user. 2019-01-28 11:44:10 +01:00
antirez 296f43b2dd ACL: create unit test. 2019-01-28 11:39:58 +01:00
Salvatore Sanfilippo 9bc46cbcc8 Merge pull request #5808 from ArkayZheng/ArkayWork
Fix the output of serverLog in replication.c
2019-01-27 12:16:14 +01:00
antirez fbc34f98c6 ACL: fix command bit check. 2019-01-26 12:51:43 +01:00
ArkayZheng 3bd953491a Fix the output bug in rename exceptions. 2019-01-25 21:48:23 +08:00
antirez 30951093cd ACL: remove the ALLCOMMANDS user flag at a safer place.
This fixes -@<category> as a side effect.
2019-01-25 13:27:33 +01:00
antirez c35ceec273 ACL: fix command ID assignment off by one error. 2019-01-25 13:07:20 +01:00
antirez 6b8582a319 ACL: remove useless NULL check. 2019-01-25 13:00:49 +01:00
antirez 72b3f7c37b Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-01-25 13:00:30 +01:00
Salvatore Sanfilippo d582ce5033 Merge pull request #5806 from soloestoy/bugfix-acl-check-key-pattern
Bugfix in ACLCheckCommandPerm
2019-01-25 13:00:17 +01:00
zhaozhao.zz 7e481d61de ACL: fix memory leak when key patterns no match 2019-01-25 19:35:18 +08:00
zhaozhao.zz 3e5ced4a7b ACL: check patterns instead of passwords in ACLCheckCommandPerm 2019-01-25 19:06:18 +08:00
antirez ed0a01e9c6 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-01-25 11:47:50 +01:00
Salvatore Sanfilippo feeb9c61c1 Merge pull request #5804 from artix75/dev
Cluster Manager: remove unused code elements
2019-01-25 11:47:45 +01:00
antirez 7a1b25de22 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2019-01-25 11:27:34 +01:00
Salvatore Sanfilippo 7466bed857 Merge pull request #5801 from soloestoy/fix-getuser-null-pointer
ACL: fix null pointer in getuser
2019-01-25 11:27:21 +01:00
antirez cecb9c6cea ACL: + and - opcode for categories implemented. 2019-01-24 18:15:46 +01:00
antirez 2910070e24 ACL: ACLSetUserCommandBitsForCategory() low level API. 2019-01-24 18:11:09 +01:00
artix 05d9bcf89f Cluster Manager: remove unused code elements 2019-01-24 17:29:44 +01:00
antirez 8a8b8a4199 ACL: populate category flags from command table. 2019-01-23 16:59:09 +01:00
zhaozhao.zz 318db2ec51 ACL: fix null pointer in getuser 2019-01-23 23:57:18 +08:00
antirez 14ea988d9c ACL: define category names and flags. 2019-01-23 16:47:29 +01:00
antirez 44dbf235c6 ACL: command categories added to the command table. 2019-01-23 12:15:10 +01:00
antirez 01a1c01bc1 ACL: improve top comments of the command table. 2019-01-23 11:21:02 +01:00
antirez 7f158ed439 ACL: mark more commands as read-only commands. 2019-01-23 11:14:39 +01:00
antirez b7f873f1c3 ACL: specify better the +@all semantics in comments. 2019-01-23 11:05:54 +01:00
antirez a885b8892b ACL: ACLUserCanExecuteFutureCommands() function added. 2019-01-23 08:14:56 +01:00
antirez 52d43a80b1 ACL: better define name, and the idea of reserved ID. 2019-01-23 08:10:57 +01:00
antirez 1e9665909d ACL: update comments in command flags. 2019-01-22 19:02:50 +01:00
antirez f1e7266c85 ACL: remove no longer existing flag from comment. 2019-01-22 17:57:20 +01:00
antirez 91a4de033a ACL: populate command table with new flags. 2019-01-22 17:57:20 +01:00
antirez 4ea77e957d ACL: review of Redis command table and new flags format.
However the code is yet not able to parse the new flags format.
2019-01-22 17:57:20 +01:00
antirez f05b44042a ACL: setuser nocommands / -@all implemented. 2019-01-22 17:57:20 +01:00
antirez 4e31893f6d ACL: setuser reset implemented. 2019-01-22 17:57:20 +01:00
antirez 10b1a9ecc6 ACL: setuser resetkeys implemented. 2019-01-22 17:57:20 +01:00
antirez b2d5ed373a ACL: specify better the command denied error. 2019-01-22 17:57:20 +01:00
antirez ec4c632302 ACL: subcommand matching. 2019-01-22 17:57:20 +01:00
antirez 355ae29c63 ACL: setting single commands ACLs implementation. 2019-01-22 17:57:20 +01:00
antirez 57f5b2dffc ACL: GetUserCommandBit() is a much better name for the function. 2019-01-22 17:57:20 +01:00
antirez 9153afb3aa ACL: user commands bitmap check refactoring for new features. 2019-01-22 17:57:20 +01:00
antirez d92b1af1e1 ACL: handle command IDs in case insensitive way. 2019-01-22 17:57:20 +01:00
Salvatore Sanfilippo d33c574ddd Merge pull request #5743 from AngusP/forever-repeat-cli
Redis CLI: Fix broken interval and repeat behaviour (incluing in cluster mode)
2019-01-22 17:29:37 +01:00
Salvatore Sanfilippo 84e3a8902e Merge pull request #5795 from zhicheng/fix-double-free-issue
fix clusterManagerGetAntiAffinityScore double free otypes
2019-01-22 17:28:45 +01:00
Salvatore Sanfilippo c291d2fc55 Merge pull request #5796 from soloestoy/fix-acl-compile-warning
ACL: fix compile warnings
2019-01-21 17:14:04 +01:00
antirez 23abe9612a Refactoring: always kill AOF/RDB child via helper functions. 2019-01-21 11:28:44 +01:00
Salvatore Sanfilippo fc5f6d1a51 Merge pull request #5798 from trevor211/fixChildInfoPipeFdLeak
Fix child info pipe fd leak when child process gets killed.
2019-01-21 11:21:12 +01:00
Salvatore Sanfilippo d8128968a9 Merge branch 'unstable' into fixChildInfoPipeFdLeak 2019-01-21 11:20:56 +01:00
antirez b3210c5a11 Remove non semantical newline space from PR #5797. 2019-01-21 11:15:48 +01:00
Salvatore Sanfilippo a010bf750e Merge pull request #5797 from trevor211/fixUpdateDictResizePolicy
Fix update dict resize policy
2019-01-21 11:14:48 +01:00
WuYunlong c00d2a334e Fix child info pipe fd leak when child process gets killed. 2019-01-21 17:48:45 +08:00
WuYunlong 9f3ebe6d59 Update dict resize policy when rdb child process gets killed. 2019-01-21 17:33:18 +08:00
WuYunlong 38e83e61fc Update dict resize policy when aof rewrite process gets killed. 2019-01-21 17:27:36 +08:00
zhaozhao.zz aee528ee66 ACL: fix compile warnings 2019-01-21 14:53:46 +08:00
Zhicheng Wei 590b97cc43 fix clusterManagerGetAntiAffinityScore double free otypes 2019-01-19 11:38:36 +08:00
antirez 4261b70f8d ACL: remove server.requirepass + some refactoring. 2019-01-18 11:49:30 +01:00
antirez b3087f5d6a ACL: fix config get requirepass. 2019-01-18 11:30:40 +01:00
antirez f1e57a86f9 ACL: implement resetpass directive and adjust test. 2019-01-18 11:26:32 +01:00
antirez 26be458e05 ACL: configure the master connection without user. 2019-01-17 18:33:36 +01:00
antirez 7e73a1fbf3 ACL: AUTH + no default user password raises an error.
This way the behavior is very similar to the past one.
This is useful in order to remember the user she probably failed to
configure a password correctly.
2019-01-17 18:30:23 +01:00
antirez a7ea1e7a2f ACL: change requirepass stop condition to use ACLs. 2019-01-17 18:22:22 +01:00
antirez b77577114b ACL: partial implementation of ACL GETUSER. 2019-01-17 18:19:04 +01:00
antirez 03c4aa01ec ACL: reimplement requirepass option in term of ACLs. 2019-01-17 18:05:43 +01:00
antirez dabd1e3bbb ACL: fix and improve ACL key checking. 2019-01-16 18:31:05 +01:00
antirez 58848500b1 ACL: create the user pattern list ASAP. 2019-01-16 13:50:00 +01:00
antirez eb6cae0b72 ACL: key matching implemented. 2019-01-16 13:39:04 +01:00
antirez 21808e3720 ACL: implement the key match opcode in ACLSetUser(). 2019-01-16 13:29:04 +01:00
antirez 2906e9c196 ACL: fix command exec check by returning. 2019-01-15 18:28:43 +01:00
antirez a2a8121ad8 ACL: the AUTH command can be always executed. 2019-01-15 18:26:44 +01:00
antirez 8cb696b953 ACL: AUTH uses users. ACL WHOAMI implemented. 2019-01-15 18:16:20 +01:00
antirez cd421f85b8 ACL: automatically authenticate the nopass default user. 2019-01-15 17:57:49 +01:00
antirez 57fa4a2f01 ACL: AUTH command new form, using the ACL subsystem. 2019-01-15 13:45:16 +01:00
antirez 1cb8998a1c ACL: nopass user setting.
This is needed in order to model the current behavior of authenticating
the connection directly when no password is set. Now with ACLs this will
be obtained by setting the default user as "nopass" user. Moreover this
flag can be used in order to create other users that do not require any
password but will work with "AUTH username <any-password>".
2019-01-15 13:16:31 +01:00
antirez 115adf0049 ACL: ability to set/remove user passwords. 2019-01-15 12:58:54 +01:00
antirez 4d8991fdf4 ACL: initial implementation of the ACL command. 2019-01-15 09:36:12 +01:00
antirez 532049a9bc ACL: ACLCheckCommandPerm() implementation WIP. 2019-01-14 18:35:21 +01:00
antirez fc85321233 RESP3: Populate new fields for the AOF fake client.
However we should remove this fake client ad-hoc creation, and replace
it with the proper call to createClient(-1), and then adjust the fake
client as we like.
2019-01-14 17:01:49 +01:00
antirez 45346c6766 ACL: fix field name typo causing segfault. 2019-01-14 16:09:29 +01:00
antirez 53053f5f30 ACL: Fix compilation by adding prototype and c->cmd fix. 2019-01-14 13:22:56 +01:00
antirez b497dbb481 ACL: ACLLCOMMAND flags. 2019-01-14 13:21:21 +01:00
antirez b9e5d4f575 ACL: Add hook in processCommand() to check the ACLs before call(). 2019-01-14 13:20:45 +01:00
antirez f4b216d32b ACL: Add skeleton for function checking ability to execute a command. 2019-01-14 13:19:50 +01:00
antirez 79e57bb854 ACL: ACLSetUser(), add allcommands in comment. 2019-01-14 13:19:42 +01:00
antirez 31994ff365 ACL: ACLSetUser(), fix flag and add allcommands +@all opcode. 2019-01-14 13:18:12 +01:00
antirez 7822181e31 ACL: implement to first trivial opcodes in ACLSetUser(). 2019-01-11 13:03:50 +01:00
antirez b2636ecdf5 ACL: avoid a radix tree lookup for the default user. 2019-01-11 11:32:41 +01:00
antirez 4890c2f5e9 ACL: modify comment from PR. 2019-01-11 11:30:09 +01:00
Salvatore Sanfilippo 2793edf8c7 Merge pull request #5769 from charsyam/feature/fix-segmentfault-when-server-starts
fix segmentfault when server start in unstable branch
2019-01-11 11:27:09 +01:00
antirez efa8afedd7 ACL: ACLSetUser(), initial ideas in comments. 2019-01-11 11:25:55 +01:00
charsyam e1ff6ef8e7 fix segmentfault when server start 2019-01-11 19:12:06 +09:00
antirez 3c538d0595 ACL: create the default user. 2019-01-11 11:02:55 +01:00
antirez 487b8fcbaa ACL: implement ACLCreateUser(). 2019-01-10 17:01:12 +01:00
antirez f3ad8fc85e ACL: implement ACLGetUserByName(). 2019-01-10 16:40:45 +01:00
antirez e75a2b8500 ACL: initialization function. 2019-01-10 16:39:32 +01:00
antirez da1c642f83 ACL: split acl.c into clear sections. 2019-01-10 16:35:55 +01:00
antirez 6771d47230 ACL: add a reference to the user in each client. 2019-01-10 16:34:13 +01:00
antirez 9633476699 ACL: improved version of the user structure. 2019-01-10 12:47:52 +01:00
antirez 3a02107a18 ACL: ACLCheckUserCredentials() next id should be static. 2019-01-09 21:47:43 +01:00
antirez cc5222bd7a ACL: use a fixed table for command IDs. 2019-01-09 21:31:29 +01:00
antirez 647a157eac ACL: set the command ID while populating the commands table. 2019-01-09 17:23:23 +01:00
antirez a571c99587 ACL: introduce the concept of command ID. 2019-01-09 17:20:47 +01:00
antirez 50927b298e ACL: fix ACLCheckUserCredentials() usage in AUTH. 2019-01-09 17:09:30 +01:00
antirez cb7b4154cd ACL: HELLO should stop if the user is not authenticated. 2019-01-09 17:00:30 +01:00
antirez 106977aaf9 ACL: refactoring of the original authentication code. 2019-01-09 17:00:30 +01:00
antirez 79c3d1355d RESP3: allow HELLO during busy script and not authenticated states. 2019-01-09 17:00:30 +01:00
antirez 26007b980a RESP3: Allow any command in RESP3 Pub/Sub mode. 2019-01-09 17:00:30 +01:00
antirez 736d38cf88 RESP3: PING should reply normally in RESP3 Pub/Sub mode.
Because in RESP3 commands can be sent in the Pub/Sub connection without
problems, so it's better if in such mode there is no exception about
PING.
2019-01-09 17:00:30 +01:00
antirez a57ea371bd RESP3: Pubsub messages in new push format if client is in RESP3 mode. 2019-01-09 17:00:30 +01:00
antirez c4bf8bf6e8 RESP3: pubsub messages API completely refactored. 2019-01-09 17:00:30 +01:00
antirez 67a6fe1dc7 RESP3: extract code to send pubsub messages into functions. 2019-01-09 17:00:30 +01:00
antirez 5190cbe08f RESP3: addReplyString() -> addReplyProto().
The function naming was totally nuts. Let's fix it as we break PRs
anyway with RESP3 refactoring and changes.
2019-01-09 17:00:30 +01:00
antirez 7db26f6aad RESP3: redis-cli support for boolean in TTY output. 2019-01-09 17:00:30 +01:00
antirez 7894b02f3c RESP3: hiredis: implement bool type. 2019-01-09 17:00:30 +01:00
antirez e936aaf5fc RESP3: verbatim reply API + DEBUG PROTOCOL support. 2019-01-09 17:00:30 +01:00
antirez a73887b0cb RESP3: DEBUG PROTOCOL: boolean types. 2019-01-09 17:00:30 +01:00
antirez fd81b96935 RESP3: Fix addReplyBool() RESP2/3 output. 2019-01-09 17:00:30 +01:00
antirez 9089aadc9d RESP3: DEBUG PROTOCOL: fix strcasecmp() check. 2019-01-09 17:00:30 +01:00
antirez 592508470d RESP3: DEBUG PROTOCOL command. Only types already supported by API. 2019-01-09 17:00:30 +01:00
antirez 229ca5b210 RESP3: fix DEBUG DIGEST-VALUE with new API. 2019-01-09 17:00:30 +01:00
antirez 092bb045fe RESP3: redis-cli: show the double as received from Redis. 2019-01-09 17:00:30 +01:00
antirez 380859ed3c RESP3: hiredis: save the original double string. 2019-01-09 17:00:30 +01:00
antirez 59448be097 RESP3: hiredis: implement null type. 2019-01-09 17:00:30 +01:00
antirez d09a0f47d1 RESP3: hiredis: fix double implementation. 2019-01-09 17:00:30 +01:00
antirez d4b34743e1 RESP3: hiredis: initial double implementation. 2019-01-09 17:00:30 +01:00
antirez 86d2b8a65f RESP3: hiredis: map and set display for TTY output. 2019-01-09 17:00:30 +01:00
antirez 1347b4eeef RESP3: hiredis: fix hiredis.c assert for new types. 2019-01-09 17:00:29 +01:00
antirez 99abef1e5d RESP3: hiredis: free map and set replies. 2019-01-09 17:00:29 +01:00
antirez 27b7e55379 RESP3: hiredis: fix read.c assert for new types. 2019-01-09 17:00:29 +01:00
antirez 5fba9d160d RESP3: hiredis updated with recent version + some RESP3 support. 2019-01-09 17:00:29 +01:00
antirez e5241b6e58 RESP3: addReplyBool() implemented. 2019-01-09 17:00:29 +01:00
antirez 75a6d12dd0 RESP3: initial implementation of the HELLO command. 2019-01-09 17:00:29 +01:00
antirez d1c79701da RESP3: fix HMGET bug introduced with RESP3 changes. 2019-01-09 17:00:29 +01:00
antirez bb98fa90bb RESP3: fix genericHgetallCommand() assert. 2019-01-09 17:00:29 +01:00
antirez f7d9fd0059 RESP3: fix zrangeGenericCommand() proto dependent array len. 2019-01-09 17:00:29 +01:00
antirez cbb6c8f978 RESP3: t_stream.c updated. 2019-01-09 17:00:29 +01:00
antirez 1916a890f7 RESP3: module.c updated. 2019-01-09 17:00:29 +01:00
antirez 38d02d0287 RESP3: latency.c updated. 2019-01-09 17:00:29 +01:00
antirez da155176f1 RESP3: hyperloglog.c updated. 2019-01-09 17:00:29 +01:00
antirez 92c9429d17 RESP3: restore the concept of null array for RESP2 compat. 2019-01-09 17:00:29 +01:00
antirez d266105a45 RESP3: add shared.nullarray for better RESP2 compat. 2019-01-09 17:00:29 +01:00
antirez 0dbdaab2a1 RESP3: addReplyNullArray() added for better RESP2 compat. 2019-01-09 17:00:29 +01:00
antirez 0972c55994 RESP3: geo.c updated. 2019-01-09 17:00:29 +01:00
antirez a97bf0c549 RESP3: blocked.c updated. 2019-01-09 17:00:29 +01:00
antirez 7ff88b2c7a RESP3: sentinel.c updated. 2019-01-09 17:00:29 +01:00
antirez 20c202c219 RESP3: bitops.c updated. 2019-01-09 17:00:29 +01:00
antirez 7872d57d27 RESP3: most null replies converted. 2019-01-09 17:00:29 +01:00
antirez 8413c6f107 RESP3: addReplyNull() added. 2019-01-09 17:00:29 +01:00
antirez a039a1bc3c RESP3: remove other pointless shared object. 2019-01-09 17:00:29 +01:00
antirez ee2564e90e RESP3: remove certain constants to spot places to fix. 2019-01-09 17:00:29 +01:00
antirez e02ab3c51e RESP3: Scripting RESP3 mode set/map protocol -> Lua conversion. 2019-01-09 17:00:29 +01:00
antirez b715bb5124 RESP3: Fix API in scripting.c leaving Lua conversions RESP2. 2019-01-09 17:00:29 +01:00
antirez bc7b1c7b51 RESP3: Use new aggregate reply API in slowlog.c. 2019-01-09 17:00:29 +01:00
antirez 0279914a41 RESP3: Use new aggregate reply API in t_set.c. 2019-01-09 17:00:29 +01:00
antirez 02c2309efc RESP3: Use new aggregate reply API in cluster.c. 2019-01-09 17:00:29 +01:00
antirez 3aebf8ab29 RESP3: Make WITHSCORES reply back with a flat array in RESP2. 2019-01-09 17:00:29 +01:00
antirez 9ed8763002 RESP3: Use new deferred len API in object.c. 2019-01-09 17:00:29 +01:00
antirez 342f336398 RESP3: bring RESP2 compatibility to previous changes. 2019-01-09 17:00:29 +01:00
antirez 76fa77f9d5 RESP3: addReply*Len() support for RESP2 backward comp. 2019-01-09 17:00:29 +01:00
antirez 6fbf149112 RESP3: put RESP version in the client structure. 2019-01-09 17:00:29 +01:00
antirez 7fe7b38743 RESP3: Use new API and types in t_hash.c. 2019-01-09 17:00:29 +01:00
antirez f5a4f6bdf4 RESP3: Use new deferred len API in dict.c. 2019-01-09 17:00:29 +01:00
antirez c3d7abb004 RESP3: Use new deferred len API in config.c. 2019-01-09 17:00:29 +01:00
antirez f10783e9fb RESP3: Use new deferred len API in t_zset.c. 2019-01-09 17:00:29 +01:00
antirez 54ccb4df9e RESP3: Use new deferred len API in t_string.c. 2019-01-09 17:00:29 +01:00
antirez 7c9f682686 RESP3: Use new deferred len API in replication.c. 2019-01-09 17:00:29 +01:00
antirez 33374b02bf RESP3: Use new deferred len API in server.c. 2019-01-09 17:00:29 +01:00
antirez 1c82be419f RESP3: Aggregate deferred lengths functions. 2019-01-09 17:00:29 +01:00
antirez da3d2db500 RESP3: Double replies and aggregate lengths initial functions. 2019-01-09 17:00:29 +01:00
Salvatore Sanfilippo 5ff8d37503 Merge pull request #5746 from UmanShahzad/old-geohash-docs
Remove documentation about geohash-int in deps repo.
2019-01-09 10:12:09 +01:00
Salvatore Sanfilippo 1eca7e34c7 Merge pull request #5729 from artix75/cluster_manager_fix_cmd
Cluster Manager del-node: use CLUSTER RESET in place of SHUTDOWN
2019-01-09 10:11:27 +01:00
Salvatore Sanfilippo f3be3d13a7 Merge pull request #5751 from chenyang8094/unstable
Update ae.c
2019-01-09 10:10:53 +01:00
chenyangyang fa4e6ba00d Update ae.c
Update comment
2019-01-06 15:01:25 +08:00
Uman Shahzad 1f66431b4a Remove documentation about geohash-int in deps repo. 2019-01-03 19:12:45 +05:00
Angus Pearson f6bd1372ab Add comment explaining negative repeat 2019-01-02 19:28:04 +00:00
Angus Pearson 314721a1ef Fix broken interval and repeat bahaviour in redis-cli (incluing cluster mode)
This addresses two problems, one where infinite (negative) repeat count is broken for all types for Redis,
and another specific to cluster mode where redirection is needed.

Now allows and works correctly for negative (i.e. -1) repeat values passed with `-r` argument to redis-cli
as documented here https://redis.io/topics/rediscli#continuously-run-the-same-command which seems to have
regressed as a feature in 95b988 (though that commit removed bad integer wrap-around to `0` behaviour).

This broken behaviour exists currently (e50458), and redis-cli will just exit immediately with repeat `-r <= 0`
as opposed to send commands indefinitely as it should with `-r < 0`

Additionally prevents a repeat * interval seconds hang/time spent doing nothing at the start before issuing
commands in cluster mode (`-c`), where the command needed to redirect to a slot on another node, as commands
where failing and waiting to be reissued but this was fully repeated before being reissued. For example,

        redis-cli -c -r 10 -i 0.5 INCR test_key_not_on_6379

Would hang and show nothing for 5 seconds (10 * 0.5) before showing

        (integer) 1
        (integer) 2
        (integer) 3
        (integer) 4
        (integer) 5
        (integer) 6
        (integer) 7
        (integer) 8
        (integer) 9
        (integer) 10

at half second intervals as intended.
2019-01-02 18:50:58 +00:00
artix 39f4a23296 Cluster Manager del-node: use CLUSTER RESET in place of SHUTDOWN
See issue #5687
2018-12-27 17:20:42 +01:00
antirez fba9c840bf Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-12-21 11:39:15 +01:00
Salvatore Sanfilippo 134c1a48c5 Merge pull request #5713 from artix75/cluster_manager_fix_cmd
Cluster Manager: enable --cluster-replace also for 'fix' command.
2018-12-19 17:34:35 +01:00
artix 853b97fd85 Cluster Manager: enable --cluster-replace also for 'fix' command. 2018-12-19 17:29:25 +01:00
Salvatore Sanfilippo 9b6ceafc00 Merge pull request #5709 from artix75/cluster_manager_fix_cmd
Fixed memory leak in clusterManagerCompareKeysValues.
2018-12-19 17:24:38 +01:00
artix 1055c6b484 Fixed memory leak in clusterManagerCompareKeysValues. 2018-12-18 18:45:10 +01:00
Salvatore Sanfilippo d54066e90f Merge pull request #5708 from artix75/cluster_manager_fix_cmd
Cluster Manager: compare key values after BUSYKEY error (migration).
2018-12-18 18:27:38 +01:00
artix 4ec37c3bd3 Cluster Manager: compare key values after BUSYKEY error (migration).
If a key exists in the target node during a migration (BUSYKEY),
the value of the key on both nodes (source and target) will be compared.
If the key has the same value on both keys, the migration will be
automatically retried with the REPLACE argument in order to override
the target's key.

If the key has different values, the behaviour will depend on such
cases:
- In case of 'fix' command, the migration will stop and the user
  will be warned to manually check the key(s).
- In other cases (ie. reshard), if the user launched the command
  with the --cluster-replace option, the migration will be
  retried with the REPLACE argument, elsewhere the migration will
  stop and the user will be warned.
2018-12-18 17:45:35 +01:00
Salvatore Sanfilippo 46baf72ddd Merge pull request #5691 from artix75/cluster_manager_fix_cmd
Cluster Manager: avoid using reply error messages to check slot status.
2018-12-12 13:39:53 +01:00
artix cc21057890 Cluster Manager: avoid using reply error messages to check slot status.
Slot assignment status is now checked by using CLUSTER SLOTS.
Furthermore, one memory leak has been fixed.
2018-12-12 13:34:43 +01:00
antirez 0ba9e0f8dd Remove debugging printf from replication.tcl test. 2018-12-12 11:55:30 +01:00
antirez b166aff1bf freeMemoryIfNeeded() small refactoring.
Related to issue #5686 and PR #5689.
2018-12-12 11:37:15 +01:00
Salvatore Sanfilippo 5daed99b5c Merge pull request #5689 from soloestoy/donot-evict-when-AOF-loading
evict: don't care about mem if loading
2018-12-12 11:29:10 +01:00
antirez 76e5519516 Crashing is too much in addReplyErrorLength().
See #5663.
2018-12-11 17:50:18 +01:00
zhaozhao.zz 19ef3890e9 evict: don't care about mem if loading
When loading data, we call processEventsWhileBlocked
to process events and execute commands.
But if we are loading AOF it's dangerous, because
processCommand would call freeMemoryIfNeeded to evict,
and that will break data consistency, see issue #5686.
2018-12-12 00:25:58 +08:00
antirez e16d2dfb5f Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-12-11 13:30:04 +01:00
antirez 7c25def9a0 stringmatchlen() fuzz test added.
Verified to be able to trigger at least #5632. Does not report other
issues.
2018-12-11 13:29:30 +01:00
antirez c30b9dc868 Fix stringmatchlen() read past buffer bug.
See #5632.
2018-12-11 13:18:52 +01:00
Salvatore Sanfilippo 5106a32402 Merge pull request #5665 from louiszhw/unstable
fix comments fault discription
2018-12-11 13:03:09 +01:00
Salvatore Sanfilippo 0a0dbf6027 Merge pull request #4264 from lsytj0413/unstable
fix a typo: craeted -> created
2018-12-11 13:02:28 +01:00
Salvatore Sanfilippo 22d9e8ce65 Merge pull request #5684 from soloestoy/ignore-multi-cmd-flags-loading-aof
multi: ignore multiState's cmd_flags when loading AOF
2018-12-11 12:57:25 +01:00
zhaozhao.zz 3ab14c56f7 fix typo 2018-12-11 19:53:54 +08:00
zhaozhao.zz e60a8ce2f2 multi: ignore multiState's cmd_flags when loading AOF 2018-12-11 19:47:36 +08:00
antirez fc4b1319d0 Reject EXEC containing write commands against RO replica.
Thanks to @soloestoy for discovering this issue in #5667.
This is an alternative fix in order to avoid both cycling the clients
and also disconnecting clients just having valid read-only transactions
pending.
2018-12-11 11:39:21 +01:00
Salvatore Sanfilippo eaaf22a4f8 Merge pull request #5681 from artix75/cluster_manager_fix_cmd
Cluster manager fix cmd
2018-12-11 08:23:54 +01:00
artix 440e8164a1 Cluster Manager:
- Multiple owners checking in 'fix'/'check' commands is now
  optional (using --cluster-search-multiple-owners).
- Updated help.
2018-12-10 18:01:18 +01:00
artix 17ff4e873f Cluster Manager:
- FixOpenSlot now correctly updates in-memory cluster configuration.
    - Improved output messages.
2018-12-10 17:43:58 +01:00
artix c2ea297052 Cluster Manager: 'fix' command now handles open slots with migrating state
in one node and importing state in multiple nodes.
2018-12-10 17:43:58 +01:00
artix b5d9d24d3b Cluster Manager: setting new slot owner is now handled atomically
in 'fix' command.
2018-12-10 17:43:58 +01:00
artix 0c7a108dc9 Cluster Manager: code cleanup. 2018-12-10 17:43:58 +01:00
artix 8b7107fd32 Cluster Manager: check/fix commands now handle multiple owners even if
all slots are covered and not open.
2018-12-10 17:39:54 +01:00
Salvatore Sanfilippo c70976ab30 Merge pull request #5626 from soloestoy/remove-useless-code
remove useless tryObjectEncoding in debug assert
2018-12-07 17:09:56 +01:00
Salvatore Sanfilippo d21f7e44ea Merge pull request #5633 from oranagra/frag_bytes_signed
fix #5580, display fragmentation and rss overhead bytes as signed
2018-12-07 17:09:10 +01:00
antirez 04d2faf7f6 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-12-07 17:07:14 +01:00
antirez 50c8580a33 Remove no longer relevant comment in processCommand(). 2018-12-07 17:06:55 +01:00
Salvatore Sanfilippo 2af190d9f8 Merge pull request #5673 from soloestoy/check-current-client-after-freememory
networking: current_client should not be NULL when trim qb_pos
2018-12-07 17:03:48 +01:00
antirez fd4bea34b5 DEBUG DIGEST-VALUE implemented. 2018-12-07 16:41:54 +01:00
antirez 439e29a38a DEBUG DIGEST refactoring: extract function to digest a value. 2018-12-07 16:30:33 +01:00
Salvatore Sanfilippo 944edff974 Merge pull request #5271 from funny-falcon/cli-migrate-replace
redis-cli reshard/rebalance: ability to force replacement on existing keys
2018-12-07 12:27:32 +01:00
zhaozhao.zz 33b1020859 networking: current_client should not be NULL when trim qb_pos 2018-12-07 19:14:33 +08:00
yura 3cd5152007 redis-cli reshard/rebalance: ability to force replacement on existing keys 2018-12-07 14:14:11 +03:00
Salvatore Sanfilippo 8b67a78dfa Merge pull request #5460 from krallin/env-password
cli: pass auth through REDISCLI_AUTH
2018-12-07 11:30:07 +01:00
Salvatore Sanfilippo 32ed397859 Merge pull request #5664 from yongman/fix-rediscli-output-format
Fix cluster call reply format readable
2018-12-07 11:28:18 +01:00
lsytj0413 c8443f9a9e fix a typo: craeted -> created 2018-12-06 11:08:59 +08:00
hdmg 35c693ff33 fix comments fault discription 2018-12-05 17:15:02 +08:00
Salvatore Sanfilippo 7b16be3fab Merge pull request #5663 from madolson/dev_monitor_crash
Fixed a serverPanic when sending an invalid command to a monitor client
2018-12-04 18:09:43 +01:00
yongman 8696f09314 Fix cluster call reply format readable 2018-12-04 17:36:42 +08:00
Madelyn Olson 38120939d1 Fixed a serverPanic when sending an invalid command to a monitor client 2018-12-04 07:17:17 +00:00
Oran Agra 1dbdef57cd fix #5580, display fragmentation and rss overhead bytes as signed
these metrics become negative when RSS is smaller than the used_memory.
This can easily happen when the program allocated a lot of memory and haven't
written to it yet, in which case the kernel doesn't allocate any pages to the process
2018-12-02 15:29:20 +02:00
zhaozhao.zz 9a4161b048 remove useless tryObjectEncoding in debug assert 2018-11-30 12:05:46 +08:00
Salvatore Sanfilippo 4808a09a4f Merge pull request #5549 from oranagra/fix_test_races
fix small test suite race conditions
2018-11-28 18:17:05 +01:00
Salvatore Sanfilippo 120daf0a04 Merge pull request #5562 from soloestoy/memory-usage-more-accurate
MEMORY command: make USAGE more accurate
2018-11-28 18:16:04 +01:00
Salvatore Sanfilippo 2648ccacb6 Merge pull request #5594 from yongman/fix-rediscli-fixcoverage
Fix choose a random master node for slot assignment
2018-11-28 18:13:04 +01:00
zhaozhao.zz cfb421fe3a MEMORY command: make USAGE more accurate
In MEMORY USAGE command, we count the key argv[2] into usage,
but the argument in command may contains free spaces because of
sdsMakeRoomFor. But the key in db never contains free spaces
because we use sdsdup when dbAdd, so using the real key to
count the usage is more accurate.
2018-11-29 01:01:51 +08:00
Salvatore Sanfilippo d4ac0287e3 Merge pull request #5572 from Weiliang-Li/patch3-util
fix comment typo in util.c
2018-11-28 17:14:21 +01:00
antirez 2d5544de8b Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-11-28 17:12:32 +01:00
Qu Chen eb1c67673d Add unit test for stream XCLAIM command. 2018-11-28 17:12:03 +01:00
Salvatore Sanfilippo 307ef33b65 Merge pull request #5597 from lamby/clarify-socket-creation-error-message
Clarify the "Creating Server TCP listening socket" error message
2018-11-28 17:06:51 +01:00
Salvatore Sanfilippo 747a4a928e Merge pull request #5598 from lamby/dont-treat-unsupported-protocols-as-fatal-errors
Don't treat unsupported protocols as fatal errors
2018-11-28 17:06:17 +01:00
Salvatore Sanfilippo bcd4fb04ad Merge pull request #5599 from devnexen/backtrace_support_on_bsd
Backtrace/register dump on BSD.
2018-11-28 17:01:33 +01:00
Salvatore Sanfilippo 8981b91c2f Merge pull request #4737 from guybe7/zlexcount_fix
Don't call sdscmp() with shared.maxstring or shared.minstring
2018-11-28 16:53:32 +01:00
antirez 4c65479744 Abort instead of crashing when loading bad stream master key.
See #5612.
2018-11-28 16:24:53 +01:00
David Carlier 131343d466 OpenBSD support.
Special treatment here as backtrace support is optional,
cannot be found via pkg-config and similar neither.
2018-11-25 08:10:26 +00:00
David Carlier 5cd50d39fb Backtrace/register dump on BSD.
FreeBSD/DragonFlyBSD does have backtrace only it does not
belong to libc.
2018-11-25 07:21:56 +00:00
Chris Lamb 99c2604541 Don't treat unsupported protocols as fatal errors
If we encounter an unsupported protocol in the "bind" list, don't
ipso-facto consider it a fatal error. We continue to abort startup if
there are no listening sockets at all.

This ensures that the lack of IPv6 support does not prevent Redis from
starting on Debian where we try to bind to the ::1 interface by default
(via "bind 127.0.0.1 ::1"). A machine with IPv6 disabled (such as some
container systems) would simply fail to start Redis after the initiall
call to apt(8).

This is similar to the case for where "bind" is not specified:

  https://github.com/antirez/redis/issues/3894

... and was based on the corresponding PR:

  https://github.com/antirez/redis/pull/4108

... but also adds EADDRNOTAVAIL to the list of errors to catch which I
believe is missing from there.

This issue was raised in Debian as both <https://bugs.debian.org/900284>
& <https://bugs.debian.org/914354>.
2018-11-23 18:01:12 +01:00
Chris Lamb be459c7e26 Clarify the "Creating Server TCP listening socket" error.
This really helps spot it in the logs, otherwise it does not look like a
warning/error. For example:

  Creating Server TCP listening socket ::1:6379: bind: Cannot assign requested address

... is not nearly as clear as:

  Could not create server TCP listening listening socket ::1:6379: bind: Cannot assign requested address
2018-11-23 17:57:01 +01:00
yongman c2fa863c55 fix typo 2018-11-23 23:51:16 +08:00
yongman 3afdde7c56 Fix choose a random master node for slot assignment 2018-11-23 16:58:55 +08:00
Salvatore Sanfilippo 5099745426 Merge pull request #5553 from devnexen/dflybsd_build_fix
DragonFlyBSD little build fix
2018-11-22 11:16:30 +01:00
Salvatore Sanfilippo 34b206057e Merge pull request #5590 from yongman/fix-cluster-fix-command
Fix command error when fixing open slots
2018-11-22 11:10:56 +01:00
yongman 1a86562f52 skip slave nodes when sending cluster setslot command 2018-11-21 23:01:35 +08:00
Salvatore Sanfilippo 59c5134218 Merge pull request #5579 from yongman/fix-rediscli-pointer-access
Fix pointer access and memory leak in redis-cli.
2018-11-20 10:38:59 +01:00
antirez 41615d418b Test: regression test for #5570. 2018-11-19 17:19:33 +01:00
antirez 9956f4f61b Stream: fix XREADGROUP history reading of deleted messages.
This commit fixes #5570. It is a similar bug to one fixed a few weeks
ago and is due to the range API to be called with NULL as "end ID"
parameter instead of repeating again the start ID, to be sure that we
selectively issue the entry with a given ID, or we get zero returned
(and we know we should emit a NULL reply).
2018-11-19 17:00:34 +01:00
antirez 195e754247 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-11-19 16:51:09 +01:00
antirez 0d7f49c3cc Test: regression test for #5577. 2018-11-19 16:50:54 +01:00
antirez cd47c66c7a Streams: fix XREADGROUP history reading when CG last_id is low.
This fixes the issue reported in #5570.
This was fixed the hard way, that is, propagating more information to
the lower level API about this being a request to read just the history,
so that the code is simpler and less likely to regress.
2018-11-19 16:41:27 +01:00
antirez f943c1d3f4 t_stream.c comment resized to 80 cols. 2018-11-19 16:26:02 +01:00
yongman 16b57b8e23 Fix pointer access and memory leak in redis-cli. 2018-11-16 17:27:27 +08:00
Weiliang Li 94bda53a8a fix comment typo in util.c
fix comment typo in util.c
2018-11-15 16:55:40 +08:00
Oran Agra 77400dde8a fix small test suite race conditions 2018-11-12 10:26:10 +02:00
David Carlier d034b38f4f DragonFlyBSD little build fix 2018-11-11 18:49:55 +00:00
Salvatore Sanfilippo eb2ec54b49 Merge pull request #5507 from devnexen/clang_build_fix_warnings
Fix clang build.
2018-11-08 12:12:28 +01:00
David Carlier 7cfbba40f0 only FreeBSD change/little warning addressing 2018-11-08 10:13:52 +00:00
antirez 8abfdc43f1 Fix cluster-replica-no-failover option name.
Thanks to @NicolasLM, see issue #5537.
2018-11-07 12:54:46 +01:00
antirez 465d1b7934 MEMORY command: make strcasecmp() conditional like the following. 2018-11-06 18:15:51 +01:00
Salvatore Sanfilippo 360eb38691 Merge pull request #4526 from itamarhaber/memory_help
Standardizes `MEMORY HELP` subcommand
2018-11-06 18:13:17 +01:00
Salvatore Sanfilippo 1d9d0b212f Merge pull request #5526 from valentino-redislabs/init-server-hz
fix short period of server.hz being uninitialised
2018-11-06 13:00:15 +01:00
Salvatore Sanfilippo 951b43eae9 Merge pull request #5520 from itamarhaber/latency_help
Adds HELP to LATENCY
2018-11-06 12:28:08 +01:00
Salvatore Sanfilippo d7fa595ed5 Merge pull request #5529 from yongman/fix-rediscli-malloc
fix zmalloc in clusterManagerComputeReshardTable
2018-11-06 12:05:24 +01:00
Salvatore Sanfilippo 88aae03968 Merge pull request #5528 from artix75/cluster_manager_fix_cmd
Cluster manager fix cmd
2018-11-06 12:05:06 +01:00
yongman 53fe09b382 fix malloc in clusterManagerComputeReshardTable 2018-11-06 10:51:19 +08:00
artix 8eb1f32c6d Cluster Manager: removed unused var. 2018-11-05 14:19:45 +01:00
artix a80bd01a4e Cluster Manager: further improvements to "fix":
- clusterManagerFixOpenSlot: ensure that the
  slot is unassigned before ADDSLOTS
- clusterManagerFixSlotsCoverage: after cold
  migration, the slot configuration
  is now updated on all the nodes.
2018-11-05 14:15:31 +01:00
artix 6ce7b41f3e Cluster Manager: fixed string parsing issue in clusterManagerGetConfigSignature 2018-11-05 14:15:22 +01:00
artix 5a64a34e85 Cluster Manager: better fix subcommand. 2018-11-05 14:15:12 +01:00
artix 18428a513a Cluster Manager: fixed typos in comments. 2018-11-05 14:15:01 +01:00
artix f90eccb9ee Cluster Manager: fixed 'DELSLOT' subcommand typo. 2018-11-05 14:14:45 +01:00
antirez 7f5f6d954e Fix XCLAIM missing entry bug.
This bug had a double effect:

1. Sometimes entries may not be emitted, producing broken protocol where
the array length was greater than the emitted entires, blocking the
client waiting for more data.

2. Some other time the right entry was claimed, but a wrong entry was
returned to the client.

This fix should correct both the instances.
2018-11-05 13:17:32 +01:00
antirez 9bfdc19946 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-11-05 13:07:14 +01:00
antirez 27cf2a7c00 Improve streamReplyWithRange() top comment. 2018-11-05 13:06:01 +01:00
valentino dcc2c9babe fix short period of server.hz being uninitialized
server.hz was uninitialized between initServerConfig and initServer.
this can lead to someone (e.g. queued modules) doing createObject,
and accessing an uninitialized variable, that can potentially be 0,
and lead to a crash.
2018-11-05 12:01:26 +02:00
Salvatore Sanfilippo 7261d1f283 Merge pull request #5521 from michael-grunder/mstime-32bit-fix
Use typedef'd mstime_t instead of time_t
2018-11-04 00:25:11 +01:00
michael-grunder 71a2f9b1b4 Use typedef'd mstime_t instead of time_t
This fixes an overflow on 32-bit systems.
2018-11-03 15:13:28 -07:00
Itamar Haber b4dfc7d502 Adds HELP to LATENCY
Signed-off-by: Itamar Haber <itamar@redislabs.com>
2018-11-02 22:58:16 +02:00
Salvatore Sanfilippo 16fa37ea0e Merge pull request #5511 from gkorland/patch-1
Fix some typos
2018-10-31 16:36:12 +01:00
Guy Korland 8e3ec37117 Fix some typos 2018-10-31 17:33:53 +02:00
antirez 44e7a8a942 Add support for Sentinel authentication.
So far it was not possible to setup Sentinel with authentication
enabled. This commit introduces this feature: every Sentinel will try to
authenticate with other sentinels using the same password it is
configured to accept clients with.

So for instance if a Sentinel has a "requirepass" configuration
statemnet set to "foo", it will use the "foo" password to authenticate
with every other Sentinel it connects to. So basically to add the
"requirepass" to all the Sentinels configurations is enough in order to
make sure that:

1) Clients will require the password to access the Sentinels instances.
2) Each Sentinel will use the same password to connect and authenticate
   with every other Sentinel in the group.

Related to #3279 and #3329.
2018-10-31 12:56:47 +01:00
antirez 7e6d1fae1b Disable protected mode in Sentinel mode.
Sentinel must be exposed, so protected mode is just an issue for users
in case Redis was started in Sentinel mode.

Related to #3279 and #3329.
2018-10-31 12:37:48 +01:00
antirez f727bf95f7 When replica kills a pending RDB save during SYNC, log it.
This logs what happens in the context of the fix in PR #5367.
2018-10-31 11:47:10 +01:00
Salvatore Sanfilippo beefadd929 Merge pull request #5367 from nUl1/fullresync-stopbgsave
Prevent RDB autosave from overwriting full resync results
2018-10-31 11:42:04 +01:00
David Carlier 41903baf9c Merge branch 'clang_build_fix_warnings' of https://github.com/devnexen/redis into clang_build_fix_warnings 2018-10-31 09:53:25 +00:00
David Carlier 266f2e8900 tweak form feedback 2018-10-31 09:53:07 +00:00
David Carlier 725fbf78ee allow flavors 2018-10-30 21:41:49 +00:00
David Carlier 2c3f9cd668 allow flavors 2018-10-30 14:38:05 +00:00
David Carlier 95423cc26c Fix clang build.
Some math functions require c11 standard.
2018-10-30 13:23:43 +00:00
antirez 3fee01a94d asyncCloseClientOnOutputBufferLimitReached(): don't free fake clients.
Fake clients are used in special situations and are not linked to the
normal clients list, freeing them will always result in Redis crashing
in one way or the other.

It's not common to send replies to fake clients, but we have one usage
in the modules API. When a client is blocked, we associate to the
blocked client object (that is safe to manipulate in a thread), a fake
client that accumulates replies. So because of this bug there was
the problem described in issue #5443.

The fix was verified to work with the provided example module. To write
a regression is very hard and unlikely to be triggered in the future.
2018-10-30 13:38:41 +01:00
Salvatore Sanfilippo a304f91816 Merge pull request #5489 from devnexen/unstable
Fix non Linux build.
2018-10-29 18:25:14 +01:00
David Carlier bcb8dcc601 needs it for the global 2018-10-26 14:12:47 +00:00
David Carlier 869612a867 Fix non Linux build.
timezone global is a linux-ism whereas it is a function under BSD.
Here a helper to get the timezone value in a more portable manner.
2018-10-26 14:02:09 +00:00
antirez 26599b4f6c Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-10-25 13:08:58 +02:00
antirez c7818eec49 Add command fingerprint comment for XSETID. 2018-10-25 13:08:48 +02:00
Salvatore Sanfilippo 7dc0315b10 Merge pull request #5454 from soloestoy/multi-oom
MULTI: OOM err if cannot free enough memory in MULTI/EXEC context
2018-10-25 13:00:19 +02:00
antirez 235645f5ae Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-10-25 11:50:15 +02:00
Salvatore Sanfilippo 1a8f69c990 Merge pull request #5459 from itamarhaber/xpending_count_underflow
A fix to XPENDING's count underflow
2018-10-25 11:50:04 +02:00
antirez f66299705b Fix XRANGE COUNT option for value of 0. 2018-10-25 11:36:24 +02:00
antirez 95770b8a52 Fix typo in streamReplyWithRange() top comment. 2018-10-24 16:28:44 +02:00
Salvatore Sanfilippo 6c69d894d6 Merge pull request #5469 from soloestoy/stat-key-miss-if-expired
if we read a expired key, misses++
2018-10-24 12:31:28 +02:00
Salvatore Sanfilippo f2d2041d60 Merge pull request #5453 from damz/pr/aof-buffer-evict
Overhead is the allocated size of the AOF buffer, not its length
2018-10-24 12:28:28 +02:00
antirez bad3e8f745 Simplify part of the #5470 patch. 2018-10-24 12:26:27 +02:00
Salvatore Sanfilippo e826aee8b2 Merge pull request #5470 from soloestoy/keys-no-trigger-expire
do not delete expired keys in KEYS command
2018-10-24 12:23:59 +02:00
antirez e0b68a22ff Use guide comments to make changes in #5462 more obvious. 2018-10-22 17:43:48 +02:00
antirez 428e595e5f Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-10-22 17:41:43 +02:00
Salvatore Sanfilippo c7cf97a957 Merge pull request #5462 from youjiali1995/fix-migrate-expired-keys
migrate: fix mismatch of RESTORE reply when some keys have expired.
2018-10-22 17:40:37 +02:00
antirez 0c98e6d015 Remove useless complexity from MSET implementation. 2018-10-22 12:24:02 +02:00
Salvatore Sanfilippo 614e880a0f Merge pull request #5451 from hujiecs/unstable
several typos fixed, optimize MSETNX to avoid unnecessary loop
2018-10-22 12:17:08 +02:00
hujie cd28b531e8 fix typo in config.c 2018-10-21 10:37:36 +08:00
hujie 0a2f8facfc fix typo 2018-10-20 12:18:56 +08:00
Thomas Orozco 91a958dbac cli: pass auth through REDISCLI_AUTH
This adds support for passing a password through a REDISCLI_AUTH
environment variable (which is safer than the CLI), which might often be
safer than passing it through a CLI argument.

Passing a password this way does not trigger the warning about passing a
password through CLI arguments, and CLI arguments take precedence over
it.
2018-10-19 19:15:14 +02:00
zhaozhao.zz 0bfb601dd3 if we read a expired key, misses++ 2018-10-19 18:16:29 +08:00
zhaozhao.zz 67e8236a06 do not delete expired keys in KEYS command 2018-10-19 18:00:57 +08:00
antirez 75f2d31bf9 Fix again stack generation on the Raspberry Pi.
The fix was removed by c8ca71d40 attempting to fix the stack generation
on ARM64, without testing if it would still work on ARM32.
Now it should work both sides.
2018-10-19 10:39:57 +02:00
youjiali1995 54fd1c83bc migrate: fix mismatch of RESTORE reply when some keys have expired. 2018-10-18 18:57:51 +08:00
Itamar Haber af29051e9d Plugs a potential underflow 2018-10-17 19:33:11 +03:00
antirez 0e3cd8fbc7 Streams: use bulk replies instead of status replies.
They play better with Lua scripting, otherwise Lua will see status
replies as "ok" = "string" which is very odd, and actually as @oranagra
reasoned in issue #5456 in the rest of the Redis code base there was no
such concern as saving a few bytes when the protocol is emitted.
2018-10-17 17:21:09 +02:00
Itamar Haber b404783fea Corrects inline documentation of syntax 2018-10-17 16:13:55 +03:00
antirez 333a39dc33 Fix conditional in XGROUP. 2018-10-17 13:00:35 +02:00
antirez 1fa6601623 Update help.h for redis-cli. 2018-10-17 12:57:21 +02:00
antirez 3d12201501 Tests for XGROUP CREATE MKSTREAM. 2018-10-17 12:11:10 +02:00
antirez 7707bd7bf3 Fix XGROUP CREATE MKSTREAM handling of . 2018-10-17 12:10:52 +02:00
antirez a4a03d8df2 Process MKSTREAM option of XGROUP CREATE at a later time.
This avoids issues with having to replicate a command that produced
errors.
2018-10-17 12:04:06 +02:00
antirez bfe8f716a3 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-10-17 11:36:03 +02:00
antirez 2e0f4eb757 XGROUP CREATE: MKSTREAM option for automatic stream creation. 2018-10-17 11:27:27 +02:00
zhaozhao.zz c0d1b24d27 MULTI: OOM err if cannot free enough memory in MULTI/EXEC context 2018-10-17 16:40:45 +08:00
Salvatore Sanfilippo 707138d5c1 Merge pull request #5333 from soloestoy/stream-commands-flag
Scripting & Streams: some commands need right flags
2018-10-17 07:50:32 +02:00
Damien Tournoud e2d397dde3 Overhead is the allocated size of the AOF buffer, not its length 2018-10-16 11:47:42 -07:00
Salvatore Sanfilippo ef9afbf96f Merge pull request #5452 from soloestoy/xsetid-lastid
Streams: rewrite empty streams with certain lastid
2018-10-16 18:56:55 +02:00
zhaozhao.zz 1e925aeb75 Streams: Tests modified XSTREAM -> XSETID 2018-10-16 23:56:26 +08:00
zhaozhao.zz 538d0ef92e Streams: rewrite empty streams with certain lastid 2018-10-16 23:47:27 +08:00
antirez 62bbb6b12e Tests modified to use XADD MAXLEN 0 + XSETID.
See #5426.
2018-10-16 17:28:56 +02:00
antirez 56c1fec40b Streams: rewrite empty streams with XADD MAXLEN 0. Use XSETID.
Related to #5426.
2018-10-16 16:48:31 +02:00
antirez 03e2232857 XSETID: accept IDs based on last entry.
Related to #5426.
2018-10-16 16:46:17 +02:00
antirez 2093b42879 Streams: XSTREAM SETID -> XSETID.
Keep vanilla stream commands at toplevel, see #5426.
2018-10-16 13:17:14 +02:00
Salvatore Sanfilippo 060cd5c941 Merge pull request #5426 from soloestoy/feature-xstream
Bugfix data inconsistency after aof rewrite, and add XSTREAM command.
2018-10-16 13:10:36 +02:00
hujiecs f119b1ef18 several typos fixed, optimize MSETNX to avoid unnecessary loop 2018-10-16 15:48:03 +08:00
Salvatore Sanfilippo d945eee8be Merge pull request #5416 from halaei/fix-timer-selected-db
fix timer context selected database
2018-10-15 13:01:58 +02:00
antirez 71d6760f48 Make comment about nack->consumer test for minidle more obvious.
Related to #5437.
2018-10-15 12:01:17 +02:00
antirez b061b05d5f Streams: use propagate_last_id itself as streamPropagateGroupID trigger.
Avoid storing the dirty value. See #5437.
2018-10-15 11:52:24 +02:00
antirez 9372eddb95 Streams: better naming: lastid_updated -> propagate_last_id.
See #5437 but also I updated a previous usage of the same var name.
2018-10-15 11:50:18 +02:00
Salvatore Sanfilippo f2929c1d34 Merge pull request #5437 from soloestoy/fix-xclaim
Fix xclaim
2018-10-15 11:47:21 +02:00
Salvatore Sanfilippo c5c1788372 Merge pull request #5440 from scadgek/patch-1
fixed typos in readme
2018-10-15 11:15:24 +02:00
antirez 05b3c803f9 redis.conf typo fixed: ingore -> ignore. 2018-10-15 11:01:37 +02:00
antirez 460e951154 Rax: radix tree updated to latest version from antirez/rax. 2018-10-13 14:17:38 +02:00
antirez ff963fade8 Test: avoid time related false positive in RESTORE test. 2018-10-13 14:17:11 +02:00
Sergey Chupov 9f32ab7e78 fixed typos in readme 2018-10-12 22:56:51 +03:00
zhaozhao.zz cae5cb86bc Streams: panic if streamID invalid after check, should not be possible. 2018-10-11 21:46:47 +08:00
zhaozhao.zz df4099c0d2 Streams: propagate lastid in XCLAIM when it has effect 2018-10-11 21:44:20 +08:00
zhaozhao.zz 929e39d1f5 Streams: XCLAIM ignore minidle if NACK is created by FORCE
Because the NACK->consumer is NULL, if idletime < minidle
the NACK does not belong to any consumer, then redis will crash
in XPENDING.
2018-10-11 21:20:49 +08:00
zhaozhao.zz fe8a6c4fba Streams: bugfix XCLAIM should propagate group name not consumer name 2018-10-11 21:12:09 +08:00
antirez 5840d1fc52 LOLWUT: capitalize Nees. 2018-10-11 13:00:55 +02:00
antirez de45608faa Test: cgroup propagation test also for NOACK variant.
Related to #5433.
2018-10-10 17:37:41 +02:00
antirez f28775a5b5 Test: consumer group last ID slave propagation test.
This is a regression for #5433.
2018-10-10 17:32:17 +02:00
antirez 110747ff92 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-10-10 12:51:39 +02:00
antirez 599e872fba Fix propagation of consumer groups last ID.
Issue #5433.
2018-10-10 12:51:02 +02:00
antirez 19b668de4c changelog.tcl: get optional argument for number of commits. 2018-10-10 11:03:36 +02:00
Salvatore Sanfilippo 8c24e72c07 Merge pull request #5428 from soloestoy/client-protected
Avoid recreate write handler for protected client.
2018-10-09 18:02:56 +02:00
zhaozhao.zz 95d9667583 Avoid recreate write handler for protected client. 2018-10-09 20:34:11 +08:00
antirez 0c1e09100f Free protected clients asynchronously.
Related to #4840.

Note that when we re-enter the event loop with aeProcessEvents() we
don't process timers, nor before/after sleep callbacks, so we should
never end calling freeClientsInAsyncFreeQueue() when re-entering the
loop.
2018-10-09 13:28:51 +02:00
antirez 1624139e91 Actually use the protectClient() API where needed.
Related to #4804.
2018-10-09 13:18:52 +02:00
antirez 25119c7069 Introduce protectClient() + some refactoring.
The idea is to have an API for the cases like -BUSY state and DEBUG
RELOAD where we have to manually deinstall the read handler.
See #4804.
2018-10-09 13:15:41 +02:00
Salvatore Sanfilippo fae1ef5ded Merge pull request #4804 from soloestoy/fix-debug-load
debug: avoid free client unexpectedly when reload & loadaof
2018-10-09 12:56:08 +02:00
antirez 0d151aa3af aof.c: improve indentation and change warning message.
Related to #5201.

I removed the !!! Warning part since compared to the other errors, a
missing EXEC is in theory a normal happening in the AOF file, at least
in theory: may happen in a differnet number of situations, and it's
probably better to don't give the user the feeling that something really
bad happened.
2018-10-09 11:51:04 +02:00
Salvatore Sanfilippo 8ce4a9b366 Merge pull request #5201 from soloestoy/abort-if-lost-exec-aof-load
AOF: discard if we lost EXEC when loading aof
2018-10-09 11:47:04 +02:00
zhaozhao.zz 197e36df46 Streams: rewrite id in XSTREAM CREATE * 2018-10-09 16:22:30 +08:00
zhaozhao.zz 9a034cd5f6 Streams: add tests for aof rewrite 2018-10-09 15:45:58 +08:00
zhaozhao.zz 0ab4126c57 Stream & AOF: rewrite stream in correct way 2018-10-09 15:40:20 +08:00
zhaozhao.zz b4d448baa9 Streams: add tests for XSTREAM command 2018-10-09 15:21:08 +08:00
zhaozhao.zz 0a968cc348 Streams: add a new command XTREAM
XSTREAM CREATE <key> <id or *> -- Create a new empty stream.
XSTREAM SETID <key> <id or $>  -- Set the current stream ID.
2018-10-09 13:11:04 +08:00
antirez 65598e3f22 Refactoring of XADD / XTRIM MAXLEN rewriting.
See #5141.
2018-10-08 12:05:22 +02:00
Salvatore Sanfilippo f654899b7f Merge pull request #5141 from soloestoy/fix-xtrim-inconsistency
Fix XTRIM and XADD with MAXLEN inconsistency
2018-10-08 12:00:00 +02:00
Salvatore Sanfilippo 0b0241012c Merge pull request #5422 from ziyasal/patch-1
Fix typo in the comment of raxSeekGreatest function
2018-10-08 11:34:42 +02:00
O(ʒ) ec7f1a0fe4 Fix typo 2018-10-07 13:37:58 +02:00
antirez 32ebb1d7d2 Fix typo in replicationCron() comment. 2018-10-05 18:30:45 +02:00
antirez d4d448d43e Fix typo in design comment of bio.c. 2018-10-05 18:29:23 +02:00
antirez 511e94cabf xclaimCommand(): fix comment typos. 2018-10-04 17:34:06 +02:00
Hamid Alaei a2ddb585ec fix timer context selected database 2018-10-04 17:02:10 +03:30
antirez de25d81b21 streamAppendItem(): Update the radix tree pointer only if changed. 2018-10-02 19:45:33 +02:00
antirez e36d334579 Listpack: optionally force reallocation on inserts.
This is useful in order to spot bugs where we fail
at updating the pointer returned by the insertion
function. Normally often the same pointer is returned,
making it harder than needed to spot bugs.

Related to #5210.
2018-10-02 19:37:36 +02:00
antirez ec656a2071 Fix printf type mismatch in genRedisInfoString(). 2018-10-02 16:26:30 +02:00
antirez 9179b87231 streamIteratorRemoveEntry(): set back lp only if pointer changed.
Most of the times the pointer will remain the same since integers of
similar size don't take more space in listpacks.

Related to #5210.
2018-10-02 16:20:40 +02:00
Salvatore Sanfilippo 132e115d29 Merge pull request #5210 from soloestoy/raxinsert-in-xdel
Streams: update listpack with new pointer in XDEL
2018-10-02 16:18:55 +02:00
Salvatore Sanfilippo f09c0fdc4e Merge pull request #5392 from soloestoy/bugfix-brpoplpush
bugfix: replace lastcmd with cmd when rewrite BRPOPLPUSH as RPOPLPUSH
2018-10-02 16:09:22 +02:00
Salvatore Sanfilippo 96b51145a3 Merge pull request #5242 from oranagra/script_mem
script cache memory in INFO and MEMORY includes both script code and overheads
2018-10-02 16:03:05 +02:00
Salvatore Sanfilippo 895fefcea4 Merge pull request #5397 from bmerry/fix-bad-zmalloc-size
Fix invalid use of sdsZmallocSize on an embedded string
2018-10-01 13:30:42 +02:00
Salvatore Sanfilippo bcb7c04e40 Merge pull request #5398 from bmerry/fix-zrealloc-accounting
Fix incorrect memory usage accounting in zrealloc
2018-10-01 13:28:53 +02:00
Salvatore Sanfilippo 2a7093e1d0 Merge pull request #5396 from oranagra/cmdstats_exec
fix #5024 - commandstats for multi-exec were logged as EXEC.
2018-10-01 13:24:49 +02:00
Salvatore Sanfilippo 86f5c8a98d Merge pull request #5400 from halaei/fix-dict-get-on-not-found
fix dict get on not found
2018-10-01 13:22:33 +02:00
Hamid Alaei 168f8b7ef6 fix dict get on not found 2018-10-01 13:57:25 +03:30
antirez 241e2e938d Fix XINFO comment for consistency. 2018-10-01 11:38:58 +02:00
Bruce Merry a2b68c02de Fix incorrect memory usage accounting in zrealloc
When HAVE_MALLOC_SIZE is false, each call to zrealloc causes used_memory
to increase by PREFIX_SIZE more than it should, due to mis-matched
accounting between the original zmalloc (which includes PREFIX size in
its increment) and zrealloc (which misses it from its decrement).

I've also supplied a command-line test to easily demonstrate the
problem. It's not wired into the test framework, because I don't know
TCL so I'm not sure how to automate it.
2018-09-30 11:49:03 +02:00
Oran Agra e531447d6c fix #5024 - commandstats for multi-exec were logged as EXEC.
this was broken a while back by ba9154d7e7bf959b002533384319a1e90545447b
the purpose of which was to fix commandstats for GEOADD
2018-09-30 12:43:11 +03:00
Bruce Merry b370259726 Fix invalid use of sdsZmallocSize on an embedded string
sdsZmallocSize assumes a dynamically allocated SDS. When given a string
object created by createEmbeddedStringObject, it calls zmalloc_size on a
pointer that isn't the one returned by zmalloc
2018-09-30 11:32:48 +02:00
antirez 3ea24c97d2 Try to avoid issues with GCC pragmas and older compilers.
See issue #5394.
2018-09-28 13:00:44 +02:00
zhaozhao.zz 2fb8b34580 bugfix: replace lastcmd with cmd when rewrite BRPOPLPUSH as RPOPLPUSH
There are two problems if we use lastcmd:

1. BRPOPLPUSH cannot be rewrited as RPOPLPUSH in multi/exec
    In mulit/exec context, the lastcmd is exec.
2. Redis will crash when execute RPOPLPUSH loading from AOF
    In fakeClient, the lastcmd is NULL.
2018-09-28 12:06:48 +08:00
Andrey Bugaevskiy 5113a00581 Move child termination to readSyncBulkPayload 2018-09-27 19:38:58 +03:00
antirez e6f36dbe8a Modules: hellodict example WIP #3: KEYRANGE. 2018-09-27 17:01:35 +02:00
antirez 4732fe03d0 Modules: Modules: dictionary API WIP #13: Compare API exported. 2018-09-27 11:46:22 +02:00
antirez 458813e132 Modules: Modules: dictionary API WIP #12: DictCompare API. 2018-09-27 11:44:25 +02:00
antirez 246d0517a8 Modules: Modules: dictionary API WIP #11: DictCompareC API. 2018-09-27 11:44:16 +02:00
antirez 427df44180 Modules: hellodict example WIP #1: GET command. 2018-09-27 11:22:43 +02:00
antirez baa2f9dbbb Modules: hellodict example WIP #1: SET command. 2018-09-26 17:52:12 +02:00
antirez 4d7e619078 Modules: remove useless defines in hellotimer.c 2018-09-26 17:42:00 +02:00
antirez 37d62f2dda Modules: fix top comment of hellotimer.c 2018-09-26 17:41:08 +02:00
antirez 0e9f302117 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-09-26 13:39:16 +02:00
antirez fee4574b1e Modules: dictionary API WIP #10: export API to modules. 2018-09-26 13:39:01 +02:00
Salvatore Sanfilippo b4cf849016 Merge pull request #5373 from gkorland/patch-5
add missing argument to function doc
2018-09-25 17:18:50 +02:00
Salvatore Sanfilippo 8122479c40 Merge pull request #5377 from Chupaka/patch-1
Fix typo in comment
2018-09-25 17:15:34 +02:00
antirez 1630079948 Modules: dictionary API WIP #9: iterator returning string object. 2018-09-25 16:49:46 +02:00
antirez a46eaab34c Modules: dictionary API WIP #8: Iterator next/prev. 2018-09-25 16:25:46 +02:00
antirez 7d375cbd91 Modules: dictionary API WIP #7: don't store the context.
Storing the context is useless, because we can't really reuse that
later. For instance in the API RM_DictNext() that returns a
RedisModuleString for the next key iterated, the user should pass the
new context, because we may run the keys of the dictionary in a
different context of the one where the dictionary was created. Also the
dictionary may be created without a context, but we may still demand
automatic memory management for the returned strings while iterating.
2018-09-25 12:58:16 +02:00
antirez fba628b4be Modules: dictionary API WIP #6: implement automatic memory management. 2018-09-25 12:45:08 +02:00
antirez 9c118be041 Modules: dictionary API work in progress #5: rename API for consistency.
By using the "C" suffix for functions getting pointer/len, we can do the
same in the future for other modules APIs that need a variant with
pointer/len and that are now accepting a RedisModuleString.
2018-09-25 12:31:46 +02:00
antirez 7091574577 Modules: change RedisModuleString API to allow NULL context.
The burden of having to always create RedisModuleString objects within a
module context was too much, especially now that we have threaded
operations and modules are doing more interesting things. The context in
the string API is currently only used for automatic memory managemnet,
so now the API was modified so that the user can opt-out this feature by
passing a NULL context.
2018-09-24 17:20:00 +02:00
antirez fb1700e148 Modules: dictionary API work in progress #4: reseek API. 2018-09-24 16:43:47 +02:00
Pavel Skuratovich b94cf34372 Fix typo in comment 2018-09-24 16:14:36 +03:00
antirez a10b1434f3 Modules: dictionary API work in progress #3: Iterator creation. 2018-09-24 11:44:49 +02:00
antirez f301984f51 Modules: dictionary API work in progress #2: Del API. 2018-09-24 11:16:58 +02:00
antirez 2cef32fd2f Modules: dictionary API work in progress #1. 2018-09-21 17:54:09 +02:00
Guy Korland 1c32aebf8c add missing argument to function doc 2018-09-21 02:46:31 +03:00
Andrey Bugaevskiy 752875e49d Prevent RDB autosave from overwriting full resync results
During the full database resync we may still have unsaved changes
on the receiving side. This causes a race condition between
synced data rename/load and the rename of rdbSave tempfile.
2018-09-19 19:58:39 +03:00
antirez 2c89c5656a Module cluster flags: use RM_SetClusterFlags() in the example. 2018-09-19 16:17:20 +02:00
antirez 950acfe8c8 Module cluster flags: add RM_SetClusterFlags() API. 2018-09-19 12:02:37 +02:00
antirez 3994918412 Module cluster flags: add hooks for NO_FAILOVER flag. 2018-09-19 11:43:37 +02:00
antirez f34da2c072 Module cluster flags: add hooks for NO_REDIRECTION flag. 2018-09-19 11:31:22 +02:00
antirez 8aa0a33dbb Module cluster flags: initial vars / defines added. 2018-09-19 11:20:52 +02:00
antirez 67210b0a6a Modules: rename the reused static client to something more general. 2018-09-18 13:22:05 +02:00
antirez 1a5e5264fa Modules: associate a fake client to timer context callback. 2018-09-18 13:19:33 +02:00
antirez 8888dcec36 Modules: associate a fake client to cluster message context callback.
Fixes #5354.
2018-09-18 13:15:40 +02:00
artix 1ece6a3995 Cluster Manager: clusterManagerFixOpenSlot now counts node's keys in slot
if node is neither migrating nor importing.
2018-09-17 16:26:46 +02:00
antirez 23b2098607 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-09-17 16:24:52 +02:00
antirez 6f9279ca14 Revert "fix repeat argument issue and reduce unnessary loop times for redis-cli."
Reverts commit 9505dd2011
since the commit introduced the very serious bug issue #5286.
2018-09-17 16:24:07 +02:00
Salvatore Sanfilippo 4af6809755 Merge pull request #5348 from gkorland/patch-2
No need to return "OK"
2018-09-17 15:42:50 +02:00
Salvatore Sanfilippo 560d210c7a Merge pull request #5352 from gkorland/patch-3
typo fix
2018-09-17 15:41:24 +02:00
antirez 4434b73f46 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-09-17 15:35:07 +02:00
antirez fde4ed5b30 Sentinel: document how to undo a renamed command. 2018-09-17 15:29:04 +02:00
Salvatore Sanfilippo 35403891ae Merge pull request #5357 from gkorland/patch-4
RedisModule_HashSet call must end with NULL
2018-09-17 15:20:10 +02:00
Guy Korland 461e06e27e Merge pull request #1 from gkorland/patch-5
Fix few typos
2018-09-17 14:15:39 +03:00
Guy Korland 572a3f9b89 Fix few typos 2018-09-17 14:13:46 +03:00
Guy Korland 1901515e92 RedisModule_HashSet call must end with NULL
Extended the RedisModule_HashSet doc to mark that each call must end with NULL
2018-09-17 13:54:56 +03:00
Guy Korland b58d8b8c8d typo fix 2018-09-16 15:21:21 +03:00
antirez 048578ab4c LOLWUT: split the command from version-specific implementations. 2018-09-14 12:35:59 +02:00
antirez 7b308a271e Slave removal: add a few forgotten aliases for CONFIG SET. 2018-09-13 21:43:15 +02:00
antirez 73b50bef3f LOLWUT: add Redis version in the output.
This creates an incentive to run the command and as a side effect
experience the art piece inside.
2018-09-13 21:36:02 +02:00
Guy Korland 75597cc3bf No need to return "OK"
No need to return "+OK" in this case since the result is an Array of all the nodes
2018-09-13 15:42:17 +03:00
antirez ec41b2150e LOLWUT: Ness -> Nees. 2018-09-13 09:03:36 +02:00
antirez ab1000bdd8 LOLWUT: Limit maximum CPU effort. 2018-09-13 08:31:55 +02:00
antirez b72f8f6dda LOLWUT: change padding conditional to a more direct one. 2018-09-13 08:29:55 +02:00
antirez 391c9b2965 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-09-13 08:08:56 +02:00
antirez ad8b827602 LOLWUT: fix crash when col < 2.
Close #5345.
2018-09-13 08:08:30 +02:00
Salvatore Sanfilippo 126d5866e1 Merge pull request #5343 from slobo/patch-1
Fix spelling descrive -> describe
2018-09-12 21:30:27 +02:00
Slobodan Mišković 88a23cedf7 Fix spelling descrive -> describe 2018-09-12 12:10:07 -07:00
antirez 658ed67342 LOLWUT: fix structure typo in comment. 2018-09-12 17:53:00 +02:00
antirez 54057e970a LOLWUT: Fix license copyright year. 2018-09-12 12:40:40 +02:00
antirez 36d3063ff9 LOLWUT: increase the translation factor. 2018-09-12 12:28:28 +02:00
antirez dfd55ed906 LOLWUT: change default size to fit a normal terminal better. 2018-09-12 12:26:33 +02:00
antirez 0a77ee90ee LOLWUT: wrap it into a proper command. 2018-09-12 11:34:10 +02:00
antirez b222cb83eb LOLWUT: draw Schotter by Georg Nees. 2018-09-12 11:16:07 +02:00
antirez 29298b3d5f LOLWUT: draw rotated squares using trivial trigonometry. 2018-09-12 10:36:02 +02:00
antirez 8cf08e5f3b LOLWUT: draw lines using Bresenham algorithm. 2018-09-11 21:15:43 +02:00
antirez 08cbc417c3 LOLWUT: Rendering of the virtual canvas to a string. 2018-09-11 17:32:57 +02:00
antirez 98a41d86d0 LOLWUT: show the output verbatim in redis-cli. 2018-09-11 16:29:31 +02:00
antirez dcc3e89174 LOLWUT: canvas structure and BSD license on top. 2018-09-11 16:09:24 +02:00
antirez 48e09d5896 LOLWUT: Emit Braille unicode according to pixel pattern. 2018-09-11 16:04:25 +02:00
Jakub Vrana b0198ab8c0 Slave removal: capitalize Replica 2018-09-11 15:32:28 +02:00
antirez e87ca3d5a8 Slave removal: remove slave from integration tests descriptions. 2018-09-11 15:32:28 +02:00
antirez edaed47358 Slave removal: remove slave from top-level tests descriptions. 2018-09-11 15:32:28 +02:00
antirez 334ce65cd8 Slave removal: remove slave from object.c. 2018-09-11 15:32:28 +02:00
antirez 2ffd1f5d24 Slave removal: remove slave from the README. 2018-09-11 15:32:28 +02:00
antirez 2c83618fae Slave removal: server.c logs fixed. 2018-09-11 15:32:28 +02:00
antirez 1365d1a03a Slave removal: remove slave from sentinel.conf when possible.
All the occurrences translated, but the ones referring to SLAVEOF
must be intact because that describe the actual Sentinel -> Redis
protocol. In theory we could send REPLICAOF to Redis instances, but
actually this would prevent Sentinel to be compatible with older Redis
instances.
2018-09-11 15:32:28 +02:00
antirez 4766fdfaaa Slave removal: replace very few things in Sentinel.
SENTINEL REPLICAS was added as an alias, in the configuration rewriting
now it uses known-replica, however all the rest is basically at API
level of logged events and messages having to do with the protocol, so
there is very little to do here compared to the Redis core itself, to
preserve compatibility.
2018-09-11 15:32:28 +02:00
antirez 3e5fa60b4f Slave removal: scripting.c logs and other stuff fixed. 2018-09-11 15:32:28 +02:00
antirez 88cef8693f Slave removal: replication.c logs fixed. 2018-09-11 15:32:28 +02:00
antirez fb6502be2f Slave removal: networking.c logs fixed. 2018-09-11 15:32:28 +02:00
antirez 1ce5c6ff00 Slave removal: blocked.c logs fixed. 2018-09-11 15:32:28 +02:00
antirez a5443a6247 Slave removal: Make obvious in redis.conf what a replica is. 2018-09-11 15:32:28 +02:00
antirez aee9d1dc3b Slave removal: slave mode -> replica mode text in redis-cli. 2018-09-11 15:32:28 +02:00
antirez cc52f2f494 Slave removal: fix typo of replicaof. 2018-09-11 15:32:28 +02:00
antirez 8347a21976 Slave removal: slave -> replica in redis.conf and output buffer option. 2018-09-11 15:32:28 +02:00
antirez 6444c30ae2 Slave removal: config.c converted + config rewriting hacks.
Aliases added for all the commands mentioning slave. Moreover CONFIG
REWRITE will use the new names, and will be able to reuse the old lines
mentioning the old options.
2018-09-11 15:32:28 +02:00
antirez 8e3686dcca Slave removal: Convert cluster.c log messages and command names. 2018-09-11 15:32:28 +02:00
antirez 88ac42587d Slave removal: redis-cli --slave -> --replica.
--slave alias remains but is undocumented, just for backward
compatibiltiy.
2018-09-11 15:32:28 +02:00
antirez 90d12c7929 Slave removal: SLAVEOF -> REPLICAOF. SLAVEOF is now an alias. 2018-09-11 15:32:28 +02:00
Salvatore Sanfilippo 756e50ccf6 Merge pull request #5307 from mesbahamin/use_geohash_defines_in_check
Use geohash limit defines in constraint check
2018-09-07 12:31:27 +02:00
Salvatore Sanfilippo 34d2c23324 Merge pull request #5239 from jeffreylovitz/cli-help-fix
CLI Help text loop verifies arg count
2018-09-07 12:28:22 +02:00
Salvatore Sanfilippo 40191136ba Merge pull request #5322 from youjiali1995/fix-sentinel-randomize
sentinel: fix randomized sentinelTimer.
2018-09-07 12:10:35 +02:00
Salvatore Sanfilippo d91e3e7965 Merge pull request #5324 from youjiali1995/fix-bio
bio: fix bioWaitStepOfType.
2018-09-07 12:06:02 +02:00
Salvatore Sanfilippo 65357f0d45 Merge pull request #5331 from Weiliang-Li/patch-redis-cli
fix usage typo in redis-cli
2018-09-06 17:41:31 +02:00
zhaozhao.zz 0b099fcd3c Scripting & Streams: some commands need right flags
xadd with id * generates random stream id

xadd & xtrim with approximate maxlen count may
trim stream randomly

xinfo may get random radix-tree-keys/nodes

xpending may get random idletime

xclaim: master and slave may have different
idletime in stream
2018-09-06 21:12:46 +08:00
Weiliang Li 15fb7210b5 fix usage typo in redis-cli 2018-09-06 13:40:05 +08:00
antirez 90504c0387 Fix scripting tests now that we default to commands repl. 2018-09-05 19:54:54 +02:00
antirez 23a5668d8f Use commands (effects) replication by default in scripts.
See issue #5250 and issue #5292 for more info.
2018-09-05 19:33:56 +02:00
antirez 1dcd1e5cfc Safer script stop condition on OOM.
Here the idea is that we do not want freeMemoryIfNeeded() to propagate a
DEL command before the script and change what happens in the script
execution once it reaches the slave. For example see this potential
issue (in the words of @soloestoy):

On master, we run the following script:

    if redis.call('get','key')
    then
        redis.call('set','xxx','yyy')
    end
    redis.call('set','c','d')

Then when redis attempts to execute redis.call('set','xxx','yyy'), we call freeMemoryIfNeeded(), and the key may get deleted, and because redis.call('set','xxx','yyy') has already been executed on master, this script will be replicated to slave.

But the slave received "DEL key" before the script, and will ignore maxmemory, so after that master has xxx and c, slave has only one key c.

Note that this patch (and other related work) was authored collaboratively in
issue #5250 with the help of @soloestoy and @oranagra.

Related to issue #5250.
2018-09-05 15:48:08 +02:00
antirez 3396fb9118 Propagate read-only scripts as SCRIPT LOAD.
See issue #5250 and the new comments added to the code in this commit
for details.
2018-09-05 15:44:33 +02:00
antirez 1ff2179d2e Don't perform eviction when re-entering the event loop.
Related to #5250.
2018-09-05 13:10:07 +02:00
youjiali1995 c7fcf97d8c bio: fix bioWaitStepOfType. 2018-09-05 16:51:13 +08:00
youjiali1995 c64f8bd56a sentinel: fix randomized sentinelTimer. 2018-09-05 10:32:18 +08:00
antirez d3465f226c Clarify why remaining may be zero in readQueryFromClient().
See #5304.
2018-09-04 13:29:27 +02:00
antirez d11a695425 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-09-04 13:26:06 +02:00
Salvatore Sanfilippo f4d332756d Merge pull request #5304 from soloestoy/fix-unexpected-readlen
networking: fix unexpected negative or zero readlen
2018-09-04 13:25:28 +02:00
Sascha Roland 24fe102997 #5299 Fix blocking XREAD for streams that ran dry
The conclusion, that a xread request can be answered syncronously in
case that the stream's last_id is larger than the passed last-received-id
parameter, assumes, that there must be entries present, which could be
returned immediately.
This assumption fails for empty streams that actually contained some
entries which got removed by xdel, ... .

As result, the client is answered synchronously with an empty result,
instead of blocking for new entries to arrive.
An additional check for a non-empty stream is required.
2018-09-04 13:13:36 +02:00
Salvatore Sanfilippo c8fbee9bb5 Merge pull request #5321 from maya-rv/patch-1
Fix typo
2018-09-04 12:58:04 +02:00
Salvatore Sanfilippo b83530ac8d Merge pull request #5315 from soloestoy/optimize-parsing-large-bulk
networking: optimize parsing large bulk greater than 32k
2018-09-04 12:49:50 +02:00
maya-rv ee5f616853 Fix typo 2018-09-04 13:32:02 +03:00
antirez 1ee4350153 Unblocked clients API refactoring. See #4418. 2018-09-03 18:39:18 +02:00
Salvatore Sanfilippo 186809f8fc Merge pull request #4418 from soloestoy/fix-multiple-unblock
fix multiple unblock for clientsArePaused()
2018-09-03 18:31:02 +02:00
antirez 700a94f653 Make pending buffer processing safe for CLIENT_MASTER client.
Related to #5305.
2018-09-03 18:17:31 +02:00
zhaozhao.zz 8ca058d77c networking: optimize parsing large bulk greater than 32k
If we are going to read a large object from network
try to make it likely that it will start at c->querybuf
boundary so that we can optimize object creation
avoiding a large copy of data.

But only when the data we have not parsed is less than
or equal to ll+2. If the data length is greater than
ll+2, trimming querybuf is just a waste of time, because
at this time the querybuf contains not only our bulk.

It's easy to reproduce the that:

Time1: call `client pause 10000` on slave.

Time2: redis-benchmark -t set -r 10000 -d 33000 -n 10000.

Then slave hung after 10 seconds.
2018-09-04 00:02:25 +08:00
zhaozhao.zz 8ef66552a1 if master is already unblocked, do not unblock it twice 2018-09-03 14:36:48 +08:00
zhaozhao.zz 81627a7505 fix multiple unblock for clientsArePaused() 2018-09-03 14:26:14 +08:00
Amin Mesbah 93fc77e39d Use geohash limit defines in constraint check
Slight robustness improvement, especially if the limit values are
changed, as was suggested in antires/redis#4291 [1].

[1] https://github.com/antirez/redis/pull/4291
2018-09-02 00:06:20 -07:00
antirez fda3dbb06c Test: processing of master stream in slave -BUSY state.
See #5297.
2018-08-31 16:45:02 +02:00
antirez 3fa8315508 After slave Lua script leaves busy state, re-process the master buffer.
Technically speaking we don't really need to put the master client in
the clients that need to be processed, since in practice the PING
commands from the master will take care, however it is conceptually more
sane to do so.
2018-08-31 16:45:02 +02:00
antirez 23338b9d59 While the slave is busy, just accumulate master input.
Processing command from the master while the slave is in busy state is
not correct, however we cannot, also, just reply -BUSY to the
replication stream commands from the master. The correct solution is to
stop processing data from the master, but just accumulate the stream
into the buffers and resume the processing later.

Related to #5297.
2018-08-31 16:45:02 +02:00
antirez adbc48bcc9 Allow scripts to timeout even if from the master instance.
However the master scripts will be impossible to kill.

Related to #5297.
2018-08-31 16:45:02 +02:00
antirez 4add364a4e Allow scripts to timeout on slaves as well.
See reasoning in #5297.
2018-08-31 16:45:01 +02:00
zhaozhao.zz f20408777a networking: fix unexpected negative or zero readlen
To avoid copying buffers to create a large Redis Object which
exceeding PROTO_IOBUF_LEN 32KB, we just read the remaining data
we need, which may less than PROTO_IOBUF_LEN. But the remaining
len may be zero, if the bulklen+2 equals sdslen(c->querybuf),
in client pause context.

For example:

Time1:

python
>>> import os, socket
>>> server="127.0.0.1"
>>> port=6379
>>> data1="*3\r\n$3\r\nset\r\n$1\r\na\r\n$33000\r\n"
>>> data2="".join("x" for _ in range(33000)) + "\r\n"
>>> data3="\n\n"
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.settimeout(10)
>>> s.connect((server, port))
>>> s.send(data1)
28

Time2:

redis-cli client pause 10000

Time3:

>>> s.send(data2)
33002
>>> s.send(data3)
2
>>> s.send(data3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.error: [Errno 104] Connection reset by peer

To fix that, we should check if remaining is greater than zero.
2018-08-31 20:02:09 +08:00
Salvatore Sanfilippo b9fa80313e Merge pull request #5268 from 0xtonyxia/fix-latency-cmd-comments2
Revise the comments of latency command.
2018-08-29 16:19:13 +02:00
Salvatore Sanfilippo 7a62176b20 Merge pull request #4216 from lamby/did-not-received-typos
Correct "did not received" -> "did not receive" typos/grammar.
2018-08-29 16:18:11 +02:00
Salvatore Sanfilippo b53571c526 Merge pull request #5282 from soloestoy/remove-duplicate-bind-in-sentinel.conf
remove duplicate bind in sentinel.conf
2018-08-29 16:13:42 +02:00
Salvatore Sanfilippo 54c7be0069 Merge pull request #5296 from soloestoy/command-script-flag2
Supplement to PR #4835, just take info/memory/command as random commands
2018-08-29 12:26:01 +02:00
zhaozhao.zz a286e88109 Supplement to PR #4835, just take info/memory/command as random commands 2018-08-29 18:23:05 +08:00
Salvatore Sanfilippo 572075e879 Merge pull request #4835 from soloestoy/command-script-flag
some commands' flags should be set correctly, issue #4834
2018-08-29 12:13:50 +02:00
zhaozhao.zz a175581eb6 some commands' flags should be set correctly, issue #4834 2018-08-29 18:07:01 +08:00
Salvatore Sanfilippo aeb2ee7ffe Merge pull request #5265 from oranagra/stabilize_tests
Fix unstable tests on slow machines.
2018-08-27 13:19:31 +02:00
antirez 28e068244d Document slave-ignore-maxmemory in redis.conf. 2018-08-27 12:34:29 +02:00
antirez 85a2a2ffa3 Make slave-ignore-maxmemory configurable. 2018-08-27 12:27:17 +02:00
antirez b0143d81f3 Introduce repl_slave_ignore_maxmemory flag internally.
Note: this breaks backward compatibility with Redis 4, since now slaves
by default are exact copies of masters and do not try to evict keys
independently.
2018-08-27 12:20:27 +02:00
antirez f3e6552ce0 Better variable meaning in processCommand(). 2018-08-27 12:17:34 +02:00
antirez d38934cc1f Re-apply rebased #2358. 2018-08-27 12:17:14 +02:00
antirez bcd0303571 Fix build errors caused by #2358. 2018-08-27 12:15:55 +02:00
zhaozhao.zz f3471ba931 remove duplicate bind in sentinel.conf 2018-08-27 12:07:24 +08:00
Salvatore Sanfilippo 22f822a340 Merge pull request #5248 from soloestoy/rewrite-brpoplpush
rewrite BRPOPLPUSH as RPOPLPUSH to propagate
2018-08-26 16:31:24 +02:00
Salvatore Sanfilippo 50e6445e91 Merge pull request #5244 from soloestoy/optimize-pipeline
pipeline: do not sdsrange querybuf unless all commands processed
2018-08-26 16:30:49 +02:00
Chris Lamb b50a6304cc Correct "did not received" -> "did not receive" typos/grammar. 2018-08-26 14:45:39 +02:00
Salvatore Sanfilippo be6e41419a Merge pull request #2992 from lamby/source-date-epoch
Use SOURCE_DATE_EPOCH over unreproducible uname + date calls.
2018-08-26 11:25:44 +02:00
Salvatore Sanfilippo aef87bfcf7 Merge pull request #2358 from lamby/config-set-maxmemory-grammar
Tidy grammar in CONFIG SET maxmemory warning.
2018-08-26 11:23:41 +02:00
Salvatore Sanfilippo f3d723f343 Merge pull request #2292 from lamby/sentinel-conf-defaults
Make some defaults explicit in the sentinel.conf for package maintainers
2018-08-26 11:19:51 +02:00
Chris Lamb 31e63f1ce7 Merge branch 'unstable' into config-set-maxmemory-grammar 2018-08-25 21:49:29 +02:00
zhaozhao.zz a7c8aa6695 networking: make setProtocolError simple and clear
Function setProtocolError just records proctocol error
details in server log, set client as CLIENT_CLOSE_AFTER_REPLY.
It doesn't care about querybuf sdsrange, because we
will do it after procotol parsing.
2018-08-23 12:21:28 +08:00
dejun.xdj fd9b7b8e97 Revise the comments of latency command. 2018-08-22 18:07:02 +08:00
Oran Agra 7723c36577 Fix unstable tests on slow machines.
Few tests had borderline thresholds that were adjusted.

The slave buffers test had two issues, preventing the slave buffer from growing:
1) the slave didn't necessarily go to sleep on time, or woke up too early,
   now using SIGSTOP to make sure it goes to sleep exactly when we want.
2) the master disconnected the slave on timeout
2018-08-21 11:46:07 +03:00
zhaozhao.zz e1966cb326 block: format code 2018-08-14 20:59:32 +08:00
zhaozhao.zz 8613125969 block: rewrite BRPOPLPUSH as RPOPLPUSH to propagate 2018-08-14 20:58:58 +08:00
zhaozhao.zz 23a50a9b79 networking: just move qb_pos instead of sdsrange in processInlineBuffer 2018-08-14 14:50:37 +08:00
zhaozhao.zz 04c3ae8020 networking: just return C_OK if multibulk processing saw a <= 0 length. 2018-08-14 13:55:30 +08:00
zhaozhao.zz 1c84c4431f adjust qbuf to 26 in test case for client list 2018-08-14 00:57:22 +08:00
zhaozhao.zz eca47868f0 pipeline: do not sdsrange querybuf unless all commands processed
This is an optimization for processing pipeline, we discussed a
problem in issue #5229: clients may be paused if we apply `CLIENT
PAUSE` command, and then querybuf may grow too large, the cost of
memmove in sdsrange after parsing a completed command will be
horrible. The optimization is that parsing all commands in queyrbuf
, after that we can just call sdsrange only once.
2018-08-14 00:43:42 +08:00
Oran Agra 20718948b7 script cache memory in INFO and MEMORY includes both script code and overheads 2018-08-13 17:36:54 +03:00
Jeffrey Lovitz ba7f792de1 CLI Help text loop verifies arg count 2018-08-12 12:47:01 -04:00
zhaozhao.zz 594d3b9023 Streams: update listpack with new pointer in XDEL 2018-08-04 01:06:53 +08:00
zhaozhao.zz d3dcee62b0 AOF: discard if we lost EXEC when loading aof 2018-08-03 23:30:34 +08:00
Salvatore Sanfilippo 4898e66aff Merge pull request #5146 from 0xtonyxia/fix-xclaim-id-parse
Streams: ID of xclaim command should start from the sixth argument.
2018-08-03 13:45:27 +02:00
Salvatore Sanfilippo abd081d9a4 Merge pull request #5151 from shenlongxing/fix-stream
Fix stream command paras
2018-08-03 13:39:32 +02:00
shenlongxing 82d4d8020f Fix stream command paras 2018-08-03 19:01:15 +08:00
antirez 53abe14ecf Fix AOF comment to report the current behavior.
Realted to #5201.
2018-08-03 12:46:06 +02:00
antirez 9d88012358 Test suite: add --loop option.
Very useful with --stop in order to catch heisenbugs.
2018-08-02 19:07:17 +02:00
antirez e76bbc5295 Test suite: new --stop option.
It pauses the test execution once the first failure is found.
2018-08-02 18:49:49 +02:00
antirez f2d987a63d Test: new sorted set skiplist order consistency.
This should be able to find new bugs and regressions about the new
sorted set update function when ZADD is used to update an element
already existing.

The test is able to find the bug fixed at 2f282aee immediately.
2018-08-02 14:15:53 +02:00
antirez bd510b68e4 Fix zslUpdateScore() edge case.
When the element new score is the same of prev/next node, the
lexicographical order kicks in, so we can safely update the node in
place only when the new score is strictly between the adjacent nodes
but never equal to one of them.

Technically speaking we could do extra checks to make sure that even if the
score is the same as one of the adjacent nodes, we can still update on
place, but this rarely happens, so probably not a good deal to make it
more complex.

Related to #5179.
2018-08-01 19:04:53 +02:00
antirez 159f8b6628 More commenting of zslUpdateScore(). 2018-08-01 18:54:15 +02:00
antirez f8d849930f Explain what's the point of zslUpdateScore() in top comment. 2018-08-01 18:53:09 +02:00
antirez 684d020fb9 Remove old commented zslUpdateScore() from source. 2018-08-01 18:50:49 +02:00
antirez 6435022be5 Optimize zslUpdateScore() as asked in #5179. 2018-08-01 18:50:31 +02:00
antirez bc61916100 zsetAdd() refactored adding zslUpdateScore(). 2018-08-01 18:27:56 +02:00
zhaozhao.zz fedb5c6353 Streams: add test cases for XADD/XTRIM maxlen 2018-08-01 10:34:29 +08:00
zhaozhao.zz 7a6234405c Streams: propagate specified MAXLEN instead of approximated
Slaves and rebooting redis may have different radix tree struct,
by different stream* config options. So propagating approximated
MAXLEN to AOF/slaves may lead to date inconsistency.
2018-08-01 10:34:12 +08:00
zhaozhao.zz faaf2dd356 Streams: reset approx_maxlen in every maxlen loop 2018-08-01 10:31:27 +08:00
zhaozhao.zz aa428375b8 Streams: XTRIM will return an error if MAXLEN with a count < 0 2018-08-01 10:31:27 +08:00
zhaozhao.zz 97716ce543 Streams: propagate original MAXLEN argument in XADD context
If we rewrite the MAXLEN argument as zero when no trimming
was performed, date between master and slave and aof will
be inconsistent, because `xtrim maxlen 0` means delete all
entries in stream.
2018-08-01 10:31:27 +08:00
antirez ac0bab8498 Streams IDs parsing refactoring.
Related to #5184.
2018-07-31 18:08:52 +02:00
Salvatore Sanfilippo 4a306f73a6 Merge pull request #5184 from 0xtonyxia/streams-id-opt
Streams: rearrange the usage of '-' and '+' IDs in stream commands.
2018-07-31 17:58:13 +02:00
antirez fdfb1fcbe2 Minor improvements to PR #5187. 2018-07-31 17:30:12 +02:00
Salvatore Sanfilippo 26be45c0f8 Merge pull request #5187 from oranagra/testsuite_improvements
Testsuite improvements
2018-07-31 17:20:09 +02:00
antirez 3ee0e176bc Cluster cron announce IP minor refactoring. 2018-07-31 17:13:03 +02:00
Salvatore Sanfilippo ff738d87d9 Merge pull request #5191 from shenlongxing/fix-memory-leak
Fix cluster-announce-ip memory leak
2018-07-31 17:11:16 +02:00
antirez 7ab75dc72e Tranfer -> transfer typo fixed. 2018-07-31 16:41:33 +02:00
Salvatore Sanfilippo c42b48d349 Merge pull request #5189 from soloestoy/refactor-dbOverwrite
refactor dbOverwrite to make lazyfree work
2018-07-31 16:40:35 +02:00
antirez 2e67ea193d Refactoring: replace low-level checks with writeCommandsDeniedByDiskError(). 2018-07-31 13:16:43 +02:00
antirez c07f1e0966 Fix writeCommandsDeniedByDiskError() inverted return value. 2018-07-31 13:14:24 +02:00
antirez e8bf6c0a94 Better top comment for writeCommandsDeniedByDiskError(). 2018-07-31 13:10:37 +02:00
antirez ca20e19104 Introduce writeCommandsDeniedByDiskError(). 2018-07-31 13:09:38 +02:00
shenlongxing 054e1c81fe Fix cluster-announce-ip memory leak 2018-07-31 16:01:44 +08:00
zhaozhao.zz f2a10a9536 refactor dbOverwrite to make lazyfree work 2018-07-31 12:07:57 +08:00
Oran Agra 65a26b6e38 test suite conveniency improvements
* allowing --single to be repeated
* adding --only so that only a specific test inside a unit can be run
* adding --skiptill useful to resume a test that crashed passed the problematic unit.
  useful together with --clients 1
* adding --skipfile to use a file containing list of tests names to skip
* printing the names of the tests that are skiped by skipfile or denytags
* adding --config to add config file options from command line
2018-07-30 19:13:15 +03:00
Salvatore Sanfilippo c4a388d4a1 Merge pull request #5153 from trevor211/fixLuaScript
Consider aof write error as well as rdb in lua script.
2018-07-30 18:10:06 +02:00
Salvatore Sanfilippo bdaa7a483c Merge pull request #5168 from rpv-tomsk/issue-5033
INFO CPU: higher precision of reported values
2018-07-30 18:06:54 +02:00
Pavel Rochnyack 327e134760 INFO CPU: higher precision of reported values
Closes: #5033
2018-07-30 18:06:22 +02:00
Salvatore Sanfilippo ec0f5c2a81 Merge pull request #5183 from gkorland/patch-1
Few typo fixes
2018-07-30 17:59:12 +02:00
antirez be48e326ca Add year in log.
User: "is there a reason why redis server logs  are missing the year in
the "date time"?"

Me: "I guess I did not imagine it would be stable enough to run for
several years".
2018-07-30 17:42:30 +02:00
dejun.xdj 8b6a07b7ba Streams: rearrange the usage of '-' and '+' IDs in stream commands. 2018-07-30 21:33:01 +08:00
dejun.xdj 26f66c4953 Streams: add mmid_supp argument in streamParseIDOrReply().
If 'mmid_supp' is set to 0, "-" and "+" will be
treated as an invalid ID.
2018-07-30 21:32:07 +08:00
Guy Korland 59f2b7c2e6 Few typo fixes 2018-07-30 16:18:56 +03:00
antirez 7e7ce3bb8f Document dynamic-hz in the example redis.conf. 2018-07-30 13:50:15 +02:00
antirez 8ef8d3f2da Make dynamic hz actually configurable. 2018-07-30 13:44:52 +02:00
antirez 6eae385b74 Control dynamic HZ via server configuration. 2018-07-30 13:37:30 +02:00
antirez 6d60062a7b Merge branch 'dynamic-hz' into unstable 2018-07-30 13:31:23 +02:00
Oran Agra 55fb237e61 add DEBUG LOG, to to assist test suite debugging 2018-07-30 12:23:51 +03:00
antirez 22829267ec Remove useless conditional from emptyDb().
Related to #4852.
2018-07-25 16:34:57 +02:00
antirez 945d779d12 Make emptyDb() change introduced in #4852 simpler to read. 2018-07-25 16:32:52 +02:00
Salvatore Sanfilippo 0e993a1d7c Merge pull request #4852 from soloestoy/optimize-flushall
optimize flushdb, avoid useless loops
2018-07-25 16:30:11 +02:00
zhaozhao.zz 31d005a3e4 optimize flushdb, avoid useless loops 2018-07-25 18:13:34 +08:00
Salvatore Sanfilippo da378a1d07 Merge pull request #5133 from soloestoy/fix-xdel-memory-leak
Streams: fix xdel memory leak
2018-07-24 18:57:05 +02:00
zhaozhao.zz 372f424399 Streams: fix xdel memory leak 2018-07-25 00:06:27 +08:00
antirez f9adc44920 Example the magic +1 in migrateCommand().
Related to #5154.
2018-07-24 17:31:43 +02:00
antirez 94f4c11942 Make changes of PR #5154 hopefully simpler. 2018-07-24 17:27:43 +02:00
Salvatore Sanfilippo 27eb7f27be Merge pull request #5154 from trevor211/optimizeMigrate
Do not migrate already expired keys.
2018-07-24 17:24:37 +02:00
antirez 88ec961edf Streams: refactoring of next entry seek in the iterator.
After #5161 the code could be made a bit more obvious for newcomers.
2018-07-24 11:07:27 +02:00
Salvatore Sanfilippo a4e994c039 Merge pull request #5161 from soloestoy/optimize-stream-iterator
Streams: skip master fileds only when we are going forward in streamIteratorGetID
2018-07-24 10:59:55 +02:00
Salvatore Sanfilippo bbdb152b2c Merge pull request #5163 from oranagra/fix_slave_buffer_test
fix slave buffer test suite false positives
2018-07-24 10:28:48 +02:00
antirez ae0232d276 string2ll(): better commenting. 2018-07-24 10:27:20 +02:00
Oran Agra c5058ebee1 fix slave buffer test suite false positives
it looks like on slow machines we're getting:
[err]: slave buffer are counted correctly in tests/unit/maxmemory.tcl
Expected condition '$slave_buf > 2*1024*1024' to be true (16914 > 2*1024*1024)

this is a result of the slave waking up too early and eating the
slave buffer before the traffic and the test ends.
2018-07-24 11:24:27 +03:00
antirez 6d54904951 Merge branch 'removeRedundantCheck' of https://github.com/dsomeshwar/redis into unstable 2018-07-24 10:24:24 +02:00
antirez 8aa5ed1e03 Restore string2ll() to original version.
See PR #5157.
2018-07-24 10:22:12 +02:00
antirez 97b08f5483 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-07-24 00:21:49 +02:00
antirez 54bbc283b1 string2ll(): remove duplicated check for special case.
Related to #5157. The PR author correctly indentified that the check was
duplicated, but removing the second one introduces a bug that was fixed
in the past (hence the duplication). Instead we can remove the first
instance of the check without issues.
2018-07-24 00:20:20 +02:00
antirez 510c2045d9 string2ll(): test for NULL pointer in all the cases. 2018-07-24 00:18:39 +02:00
Salvatore Sanfilippo 6264e81b85 Merge pull request #5160 from oranagra/fix_zmalloc_usable
fix recursion typo in zmalloc_usable
2018-07-24 00:07:59 +02:00
antirez 1ecfe03852 Change 42 to 1000 as warning level for cached scripts.
Related to #4883.
2018-07-23 18:44:58 +02:00
Salvatore Sanfilippo a5f07a06f4 Merge pull request #4883 from itamarhaber/lua_scripts-in-info-memory
Adds memory information about the scripts' cache to INFO
2018-07-23 18:43:05 +02:00
antirez 3d32af7ec6 Dynamic HZ: adapt cron frequency to number of clients. 2018-07-23 14:21:04 +02:00
antirez 6bfbad38af Dynamic HZ: separate hz from the configured hz.
This way we can remember what the user configured HZ is, but change the
actual HZ dynamically if needed in the dynamic HZ feature
implementation.
2018-07-23 14:13:58 +02:00
Itamar Haber 527786c0f3 Adds Lua overheads to MEMORY STATS, smartens the MEMORY DOCTOR 2018-07-22 21:16:00 +03:00
zhaozhao.zz 8e01dbe90b Streams: skip master fileds only when we are going forward in streamIteratorGetID 2018-07-22 17:52:09 +08:00
Oran Agra 359705c08d fix recursion typo in zmalloc_usable 2018-07-22 10:17:35 +03:00
dsomeshwar af41c94526 removing redundant check 2018-07-21 23:42:08 +05:30
WuYunlong 89de21d6a6 Do not migrate already expired keys. 2018-07-21 10:00:32 +08:00
WuYunlong ea2a56acf5 Consider aof write error as well as rdb in lua script. 2018-07-21 08:48:51 +08:00
antirez ebce3647df Top comment clientsCron(). 2018-07-20 09:46:18 +02:00
antirez ee9b50b0cb Clarify that clientsCronTrackExpansiveClients() indexes may jump ahead. 2018-07-20 09:36:48 +02:00
antirez 502fe74232 Rename INFO CLIENT max buffers field names for correctness.
They are actually delayed a few seconds, so let's call them "recent".
2018-07-19 17:38:20 +02:00
antirez e42fe2628e Fix wrong array index variable in getExpansiveClientsInfo(). 2018-07-19 17:34:15 +02:00
antirez 28557ee271 Change INFO CLIENTS sections to report pre-computed max/min client buffers. 2018-07-19 17:16:19 +02:00
dejun.xdj 950bff839f Streams: ID of xclaim command starts from the sixth argument. 2018-07-19 20:48:08 +08:00
antirez b3d4d162b3 Rename var in clientsCronTrackExpansiveClients() for clarity. 2018-07-19 13:59:13 +02:00
antirez 2373eaccf1 Implement a function to retrieve the expansive clients mem usage. 2018-07-19 13:58:04 +02:00
antirez 9c87d5cabe clientsCronTrackExpansiveClients() actual implementation. 2018-07-19 13:54:20 +02:00
antirez 363edba137 clientsCronTrackExpansiveClients() skeleton and ideas. 2018-07-19 13:49:00 +02:00
antirez 7401f4b7cf Make vars used only by INFO CLIENTS local to the block.
Related to #4727.
2018-07-19 12:23:05 +02:00
Salvatore Sanfilippo 8b638d90be Merge pull request #4727 from kingpeterpaule/redis-fix-info-cli
move getClientsMaxBuffers func into  info clients command
2018-07-19 12:21:35 +02:00
antirez be0266ab8d Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-07-18 17:41:37 +02:00
antirez 8f94829fa3 In addReplyErrorLength() only panic when replying to slave.
See #5135 for more context.
2018-07-18 17:41:16 +02:00
antirez 84d17a5f72 Refine comment in addReplyErrorLength() about replying to masters/slaves.
See #5135 for some context.
2018-07-18 17:40:07 +02:00
Salvatore Sanfilippo 4ff114bcfa Merge pull request #5138 from oranagra/improve_defrag_test
make active defrag test more stable
2018-07-18 11:05:03 +02:00
Oran Agra 0a4d1f114e make active defrag test more stable
on slower machines, the active defrag test tended to fail.
although the fragmentation ratio was below the treshold, the defragger was
still in the middle of a scan cycle.

this commit changes:
- the defragger uses the current fragmentation state, rather than the cache one
  that is updated by server cron every 100ms. this actually fixes a bug of
  starting one excess scan cycle
- the test lets the defragger use more CPU cycles, in hope that the defrag
  will be faster, but also give it more time before we give up.
2018-07-18 10:16:33 +03:00
Salvatore Sanfilippo 7050029b5d Merge pull request #5122 from trevor211/allowWritesWhenAofDisabled
Accept write commands if persisting is disabled
2018-07-17 18:08:46 +02:00
Salvatore Sanfilippo b8d6bdf224 Merge pull request #4237 from aspirewit/update-comment
Update the comment
2018-07-17 18:04:55 +02:00
antirez cc69607936 Streams: better error when $ is given with XREADGROUP. 2018-07-17 17:54:10 +02:00
Salvatore Sanfilippo 7fd0720ea1 Merge pull request #5136 from 0xtonyxia/fix-xread-id-parse
Fix xreadgroup with '$' ID.
2018-07-17 17:50:31 +02:00
antirez 919c89d5ee Panic when we are sending an error to our master/slave.
Related to #5135, see discussion there.
2018-07-17 17:42:30 +02:00
Salvatore Sanfilippo 289149412b Merge pull request #5135 from oranagra/rare_repl_corruption
fix rare replication stream corruption with disk-based replication
2018-07-17 17:33:58 +02:00
dejun.xdj 5b33aee802 Streams: remove meaningless if condition.
It's already checked if xreadgroup is set and groupname is NULL.
2018-07-17 18:23:47 +08:00
dejun.xdj 6eb1f8add6 Streams: return an error message if using xreadgroup with '$' ID.
Redis will always return an empty result when '$' ID is specified
with xreadgroup command, it's meaningless.
2018-07-17 18:19:10 +08:00
Oran Agra e8b73732d9 fix rare replication stream corruption with disk-based replication
The slave sends \n keepalive messages to the master while parsing the rdb,
and later sends REPLCONF ACK once a second. rarely, the master recives both
a linefeed char and a REPLCONF in the same read, \n*3\r\n$8\r\nREPLCONF\r\n...
and it tries to trim two chars (\r\n) from the query buffer,
trimming the '*' from *3\r\n$8\r\nREPLCONF\r\n...

then the master tries to process a command starting with '3' and replies to
the slave a bunch of -ERR and one +OK.
although the slave silently ignores these (prints a log message), this corrupts
the replication offset at the slave since the slave increases the replication
offset, and the master did not.

other than the fix in processInlineBuffer, i did several other improvments
while hunting this very rare bug.

- when redis replies with "unknown command" it includes a portion of the
  arguments, not just the command name. so it would be easier to understand
  what was recived, in my case, on the slave side,  it was -ERR, but
  the "arguments" were the interesting part (containing info on the error).
- about a year ago i added code in addReplyErrorLength to print the error to
  the log in case of a reply to master (since this string isn't actually
  trasmitted to the master), now changed that block to print a similar log
  message to indicate an error being sent from the master to the slave.
  note that the slave is marked as CLIENT_SLAVE only after PSYNC was received,
  so this will not cause any harm for REPLCONF, and will only indicate problems
  that are gonna corrupt the replication stream anyway.
- two places were c->reply was emptied, and i wanted to reset sentlen
  this is a precaution (i did not actually see such a problem), since a
  non-zero sentlen will cause corruption to be transmitted on the socket.
2018-07-17 12:51:49 +03:00
antirez 515c541079 dict.c: remove a few trailing spaces. 2018-07-17 10:39:47 +02:00
Salvatore Sanfilippo b51b198697 Merge pull request #5128 from kingpeterpaule/remove-one-loop-in-freeMemoryIfNeeded
remove ineffective loop in dictGetSomeKeys.
2018-07-17 10:38:55 +02:00
Salvatore Sanfilippo 49a0d4977c Merge pull request #5113 from 0xtonyxia/using-compare-func-instead
Streams: using streamCompareID() instead of direct compare.
2018-07-16 18:34:35 +02:00
Salvatore Sanfilippo fc715bb56c Merge pull request #5127 from oranagra/sds_req_type
bugfix in sdsReqType creating 64bit sds headers on 32bit systems
2018-07-16 18:32:14 +02:00
antirez 5b632ffd58 Hopefully improve commenting of #5126.
Reading the PR gave me the opportunity to better specify what the code
was doing in places where I was not immediately sure about what was
going on. Moreover I documented the structure in server.h so that people
reading the header file will immediately understand what the structure
is useful for.
2018-07-16 17:56:54 +02:00
Salvatore Sanfilippo bfab478873 Merge pull request #5126 from oranagra/slave_buf_memory_2
slave buffers were wasteful and incorrectly counted causing eviction
2018-07-16 17:45:50 +02:00
Salvatore Sanfilippo 96f847fa8f Merge pull request #5132 from soloestoy/propagate-xdel-correctly
Streams: correctly propagate xdel if needed
2018-07-16 16:04:32 +02:00
Oran Agra 1f2ed12d07 slave buffers were wasteful and incorrectly counted causing eviction
A) slave buffers didn't count internal fragmentation and sds unused space,
   this caused them to induce eviction although we didn't mean for it.

B) slave buffers were consuming about twice the memory of what they actually needed.
- this was mainly due to sdsMakeRoomFor growing to twice as much as needed each time
  but networking.c not storing more than 16k (partially fixed recently in 237a38737).
- besides it wasn't able to store half of the new string into one buffer and the
  other half into the next (so the above mentioned fix helped mainly for small items).
- lastly, the sds buffers had up to 30% internal fragmentation that was wasted,
  consumed but not used.

C) inefficient performance due to starting from a small string and reallocing many times.

what i changed:
- creating dedicated buffers for reply list, counting their size with zmalloc_size
- when creating a new reply node from, preallocate it to at least 16k.
- when appending a new reply to the buffer, first fill all the unused space of the
  previous node before starting a new one.

other changes:
- expose mem_not_counted_for_evict info field for the benefit of the test suite
- add a test to make sure slave buffers are counted correctly and that they don't cause eviction
2018-07-16 16:43:42 +03:00
zhaozhao.zz cdbe9d2c23 Streams: correctly propagate xdel if needed 2018-07-16 20:48:07 +08:00
antirez df8d210c7e Add a few comments to streamIteratorRemoveEntry(). 2018-07-16 12:41:55 +02:00
Salvatore Sanfilippo 6c92429b55 Merge pull request #5131 from soloestoy/optimize-xdel
Streams: free lp if all elements are deleted
2018-07-16 12:39:38 +02:00
antirez 1b9d47274a Modify XINFO field from last-id to last-generated-id.
Related to #5129.
2018-07-16 12:25:26 +02:00
Salvatore Sanfilippo ff84677102 Merge pull request #5129 from soloestoy/xinfo-show-last-id
Streams: show last id for streams and groups
2018-07-16 12:24:14 +02:00
zhaozhao.zz 97335c7baa Streams: free lp if all elements are deleted 2018-07-16 15:57:41 +08:00
paule 3fb051b8da Update dict.c
change coding style.
2018-07-16 14:29:59 +08:00
zhaozhao.zz 3c2425daf5 Streams: show last id for streams and groups 2018-07-16 13:24:54 +08:00
peterpaule 0ef23ec9f1 remove one ineffective loop in dictGetSomeKeys. 2018-07-16 11:28:22 +08:00
Oran Agra 72edb9426b bugfix in sdsReqType creating 64bit sds headers on 32bit systems 2018-07-15 18:24:18 +03:00
dejun.xdj 369a6bd1e9 Streams: using streamCompareID() instead of direct compare in block.c. 2018-07-14 15:03:05 +08:00
dejun.xdj 80257dfa77 Streams: add streamCompareID() declaration in stream.h. 2018-07-14 15:02:24 +08:00
dejun.xdj 4b2b9999f0 Streams: using streamCompareID() instead of direct compare. 2018-07-14 20:34:06 +08:00
WuYunlong d01a2d759e Accept write commands if persisting is disabled,
event if we do have problems persisting on disk
previously.
2018-07-14 09:06:23 +08:00
Salvatore Sanfilippo a3e3b81db4 Merge pull request #5120 from andrewsensus/leap-year-comment-patch-1
update leap year comment
2018-07-13 18:18:45 +02:00
antirez 4b8c53d243 Test: XDEL fuzz testing. Remove and check stage. 2018-07-13 17:58:17 +02:00
antirez 04bdc2a9f9 Test: fix lshuffle by providing the "K" combinator. 2018-07-13 17:52:39 +02:00
antirez c3fadab01e Test: add lshuffle in the Tcl utility functions set. 2018-07-13 17:51:03 +02:00
antirez 9d5178360d Test: XDEL fuzz testing, stream creation. 2018-07-13 17:47:26 +02:00
andrewsensus c16cdea67e update leap year comment 2018-07-13 09:46:30 -06:00
antirez b25238baa9 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-07-13 17:41:10 +02:00
antirez de5d4dbed6 Test: XDEL basic test. 2018-07-13 17:40:48 +02:00
Salvatore Sanfilippo e7e70d9272 Merge pull request #5119 from trevor211/fixSlowlogConfig
Fix slowlog config
2018-07-13 17:35:15 +02:00
WuYunlong f8c4a65d99 Fix config set slowlog-log-slower-than and condition in createLatencyReport 2018-07-13 17:53:55 +08:00
WuYunlong 89e8472e0f Add test in slowlog.tcl 2018-07-13 17:51:06 +08:00
artix b81d389020 Cluster Manager: more checks on --cluster-weight option. 2018-07-13 10:51:58 +02:00
artix e2230c1c15 Redis-trib deprecated: it no longer works and it
outputs a warning to the user.
2018-07-13 10:51:58 +02:00
artix d000b8f809 Cluster Manager: auth support (-a argument). 2018-07-13 10:51:58 +02:00
Salvatore Sanfilippo e935c6dd5c Merge pull request #5115 from shenlongxing/patch-1
Delete unused role checking.
2018-07-12 19:07:40 +02:00
Shen Longxing 4cb47f2e97 Delete unused role checking.
When check rdb file, it is unnecessary to check role.
2018-07-12 21:21:37 +08:00
Salvatore Sanfilippo 26e7d33b2c Merge pull request #4820 from charpty/wip-serverc-simplify
Remove unnecessary return statements
2018-07-12 13:42:15 +02:00
antirez fe41bc4ab1 Add regression test for #5111. 2018-07-12 13:35:17 +02:00
antirez 5d98cedeb1 Streams: when re-delivering because of SETID, reset deliveries counter.
We don't want to increment the deliveries here, because the sysadmin
reset the consumer group so the desire is likely to restart processing,
and having the PEL polluted with old information is not useful but
probably confusing.

Related to #5111.
2018-07-12 13:19:39 +02:00
antirez eaf10517ec Simplify duplicated NACK #5112 fix.
We don't really need to distinguish between the case the consumer is the
same or is a different one.
2018-07-12 13:15:54 +02:00
Salvatore Sanfilippo 27229d1511 Merge pull request #5112 from 0xtonyxia/fix-xreadgroup-crash-after-setid
Streams: fix xreadgroup crash after xgroup SETID is sent.
2018-07-12 13:04:22 +02:00
Salvatore Sanfilippo f094632d0c Merge pull request #5114 from oranagra/defrag_32
Active defrag fixes for 32bit builds (again)
2018-07-11 22:07:46 +02:00
Oran Agra b18e55c498 Active defrag fixes for 32bit builds (again)
* overflow in jemalloc fragmentation hint to the defragger
2018-07-11 16:09:00 +03:00
Salvatore Sanfilippo c765962f8c Merge pull request #4967 from JingchengLi/unstable
fix repeat argument issue and reduce unnessary loop times for redis-cli.
2018-07-10 15:13:20 +02:00
tengfeng 22c533560c fix repeat argument issue and reduce unnessary loop times for redis-cli. 2018-07-10 20:58:01 +08:00
antirez 1a5657ba73 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-07-10 12:06:44 +02:00
antirez 01d2f98f1e Streams: fix typo "consumer". 2018-07-10 12:04:31 +02:00
antirez 39ae0c9d43 Streams: fix new XREADGROUP sync logic. 2018-07-10 12:02:03 +02:00
antirez a16c80b107 Streams: make blocking for > a truly special case.
To simplify the semantics of blocking for a group, this commit changes
the implementation to better match the description we provide of
conusmer groups: blocking for > will make the consumer waiting for new
elements in the group. However blocking for any other ID will always
serve the local history of the consumer.

However it must be noted that the > ID is actually an alias for the
special ID ms/seq of UINT64_MAX,UINT64_MAX.
2018-07-10 11:34:17 +02:00
antirez 498f3e67a7 Streams: send an error to consumers blocked on non-existing group.
To detect when the group (or the whole key) is destroyed to send an
error to the consumers blocked in such group is a problem, so we leave
the consumers listening, the sysadmin is free to create or destroy
groups assuming she/he knows what to do. However a client may be blocked
in a given consumer group, that is later destroyed. Then the stream
receives new elements. In that case there is no sane behavior to serve
the consumer... but to report an error about the group no longer
existing.

More about detecting this synchronously and why it is not done:

1. Normally we don't do that, we leave clients blocked for other data
types such as lists.

2. When we free a stream object there is no longer information about
what was the key it was associated with, so while destroying the
consumer groups we miss the info to unblock the clients in that moment.

3. Objects can be reclaimed in other threads where it is no longer safe
to do client operations.
2018-07-10 11:19:06 +02:00
antirez feef1367c6 Streams: fix unblocking logic into a consumer group.
When a client blocks for a consumer group, we don't know the actual ID
we want to be served: other clients blocked in the same consumer group
may be served first, so the consumer group latest delivered ID changes.
This was not handled correctly, all the clients in the consumer group
were unblocked without data but the first.
2018-07-10 11:11:41 +02:00
dejun.xdj 4ba944c3eb Streams: fix xreadgroup crash after xgroup SETID is sent.
For issue #5111.
2018-07-10 16:26:13 +08:00
Salvatore Sanfilippo e4d7cbfe2c Merge pull request #5095 from 0xtonyxia/fix-indentation
Fix indentation.
2018-07-09 13:56:39 +02:00
dejun.xdj 10255ac477 Bugfix: PEL is incorrect when consumer is blocked using xreadgroup with NOACK option.
Save NOACK option into client.blockingState structure.
2018-07-09 13:40:29 +02:00
antirez 28d5d87741 redis-cli: fix #4990 additional argument in help. 2018-07-09 13:36:47 +02:00
antirez e938b8a1dd redis-cli: fix #5096 double error message. 2018-07-09 13:25:48 +02:00
antirez 375589d028 redis-cli: cliConnect() flags CC_FORCE and CC_QUIET.
We need CC_QUIET in order to fix #5096 by silently failing if needed.
2018-07-09 13:24:30 +02:00
Amit Dey 2a28f06a75 fixing broken link in CONTRIBUTING
http://help.github.com/send-pull-requests/ 
is no longer supported

this change modifies the link to the working one
https://help.github.com/articles/creating-a-pull-request/
2018-07-09 13:14:57 +02:00
dejun.xdj 5335463d44 CLIENT UNBLOCK: fix client unblock help message. 2018-07-09 13:03:57 +02:00
minkikim89 c37742de0e fix whitespace in redis-cli.c 2018-07-09 12:59:13 +02:00
WuYunlong ad57103fed fix compile warning in addReplySubcommandSyntaxError 2018-07-09 12:57:12 +02:00
antirez aefa752cf9 Use nolocks_localtime() for safer logging. 2018-07-04 16:50:22 +02:00
antirez 7604f6792f Cache timezone and daylight active flag for safer logging.
With such information will be able to use a private localtime()
implementation serverLog(), which does not use any locking and is both
thread and fork() safe.
2018-07-04 16:45:00 +02:00
antirez 3a6894cdd2 Localtime: clarify is_leap_year() working with comments. 2018-07-04 15:50:41 +02:00
antirez e522dd5788 Localtime: fix comment about leap year. 2018-07-04 15:16:54 +02:00
antirez 6d1b29d7c6 Localtime: fix daylight saving adjustment. Use * not +. 2018-07-04 15:16:20 +02:00
antirez fbd7290478 Localtime: fix daylight time documentation and computation. 2018-07-04 14:09:07 +02:00
dejun.xdj aa62b5ebe9 Fix indentation. 2018-07-04 20:04:06 +08:00
antirez d21c58867e Localtime: add a test main() function to check the output. 2018-07-04 13:46:31 +02:00
antirez a72f9c1931 Localtime: day of month is 1 based. Convert from 0 based "days". 2018-07-04 13:35:40 +02:00
antirez 5caae0e6e9 Localtime: fix timezone adjustment. 2018-07-04 13:35:15 +02:00
antirez ecba1a2fd4 Localtime: compute year, month and day of the month. 2018-07-04 13:25:55 +02:00
antirez bb802969d7 Localtime: basics initial calculations. Year missing. 2018-07-04 13:16:53 +02:00
antirez e8f72804f5 Localtime function skeleton and file added. 2018-07-04 12:56:12 +02:00
Jack Drogon df7bafeb44 Fix typo 2018-07-03 18:19:46 +02:00
antirez 57dc5832a7 Clarify the pending_querybuf field of clients. 2018-07-03 13:25:41 +02:00
antirez dc6c3bfe71 Improve style of PR #5084. 2018-07-03 13:23:29 +02:00
Salvatore Sanfilippo 4bcdfb0c5d Merge pull request #5084 from chendq8/pending-querybuf
limit the size of pending-querybuf in masterclient
2018-07-03 13:19:37 +02:00
antirez 32fc058177 Set repl_down_since to zero on state change.
PR #5081 fixes an "interesting" bug about Redis Cluster failover but in
general about the updating of repl_down_since, that is used in order to
count the time a slave was left disconnected from its master.

While the fix provided resolves the specific issue, in general the
validity of repl_down_since is limited to states that are different
than the state CONNECTED, and the disconnected time is set when the
state is DISCONNECTED. However from CONNECTED to other states, the state
machine must always go to DISCONNECTED first. So it makes sense to set
the field to zero (since it is meaningless in that context) when the
state is set to CONNECTED.
2018-07-03 12:42:14 +02:00
Salvatore Sanfilippo c8110e814c Merge pull request #5081 from trevor211/fixClusterFailover
cluster failover bug
2018-07-03 12:36:53 +02:00
chendianqiang 0a32c93dd5 Merge branch 'unstable' into pending-querybuf 2018-07-03 10:07:26 +08:00
antirez 9f30306d44 addReplySubSyntaxError() renamed to addReplySubcommandSyntaxError(). 2018-07-02 18:49:34 +02:00
Salvatore Sanfilippo e7b480369b Merge pull request #4998 from itamarhaber/module_command_help
Module command help
2018-07-02 18:46:56 +02:00
Salvatore Sanfilippo b077de1862 Merge pull request #5071 from akshaynagpal/patch-2
Added link to Google Group
2018-07-02 18:41:58 +02:00
Salvatore Sanfilippo 5c3a0414d0 Merge pull request #5068 from shenlongxing/fix-rename-command
fix empty string for sentinel rename-command
2018-07-02 18:40:35 +02:00
Salvatore Sanfilippo dc30938cbc Merge pull request #5090 from trevor211/test_helper_tcl
fix tests/test_helper.tcl with --wait-server option.
2018-07-02 17:51:04 +02:00
WuYunlong a14da4dfe3 fix tests/test_helper.tcl with --wait-server option.
Issue #5063 added --wait-server option, but can not work.
2018-07-02 19:26:52 +08:00
chendianqiang 2dc66b6bcc limit the size of pending-querybuf in masterclient 2018-07-01 14:43:53 +08:00
WuYunlong cfa0f8850d fix server.repl_down_since resetting, so that slaves could failover
automatically as expected.
2018-06-30 09:39:08 +08:00
WuYunlong f50f714c92 cluster.tcl: Add master consecutively down test. 2018-06-30 09:29:35 +08:00
antirez 56750fbaed Change CLIENT LIST TYPE help string.
Making it more similar to KILL.
2018-06-29 18:03:00 +02:00
Salvatore Sanfilippo 3314ac96e3 Merge pull request #5075 from soloestoy/client-list-types
FEATURE: implements client list type option
2018-06-29 18:00:32 +02:00
Salvatore Sanfilippo 7f0c97febf Merge pull request #5074 from soloestoy/fix-compile-warning
fix some compile warnings
2018-06-29 17:56:47 +02:00
Salvatore Sanfilippo 81597e4e64 Merge pull request #5076 from 0xtonyxia/add-no-auth-warning-option
Add no auth warning option
2018-06-29 13:32:58 +02:00
dejun.xdj 4535fd3b8b Check if password is used on command line interface. 2018-06-28 19:09:22 +08:00
dejun.xdj 2171b9622d Fix trailing white space. 2018-06-28 18:47:28 +08:00
dejun.xdj e9feeb822a Fix code format issue. 2018-06-28 18:39:42 +08:00
dejun.xdj 47308e7a14 Don't output password warning message when --no-auth-warning is used. 2018-06-28 18:38:24 +08:00
dejun.xdj a9ec9aa0ae Avoid -Woverlength-strings compile warning.
Using another fprintf call to output the rest help message.
2018-06-28 18:22:06 +08:00
antirez 4ff8dc478f Rax library updated (node callback). 2018-06-28 12:19:06 +02:00
dejun.xdj e8a7af858a Add --no-auth-warning help message. 2018-06-28 18:17:04 +08:00
zhaozhao.zz 483f7d3222 clients: add type option for client list 2018-06-28 17:43:05 +08:00
zhaozhao.zz eb6fa7e4af clients: show pubsub flag in client list 2018-06-28 17:28:38 +08:00
zhaozhao.zz 7a04c30907 fix some compile warnings 2018-06-28 17:22:59 +08:00
Akshay Nagpal cd5e9a5fd2 Added link to Google Group 2018-06-27 20:08:39 -04:00
antirez ec9e232d87 Make CLIENT HELP output nicer to the eyes. 2018-06-28 00:21:32 +02:00
antirez dee970724d Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-28 00:17:21 +02:00
antirez 60ade804fe Add unblock in CLIENT HELP. 2018-06-28 00:17:10 +02:00
shenlongxing e38f95a5ad fix empty string for sentinel rename-command 2018-06-28 01:08:55 +08:00
Salvatore Sanfilippo 5b7d5add43 Merge pull request #5066 from oranagra/defrag_jemalloc5_fix
minor fix in creating a stream NACK for rdb and defrag tests
2018-06-27 18:56:49 +02:00
Salvatore Sanfilippo 11b19b264f Merge pull request #5067 from mpaltun/mpaltun-doc-fix
Streams documentation fix
2018-06-27 18:54:27 +02:00
antirez c14090b8f4 CLIENT UNBLOCK: support unblocking by error. 2018-06-27 18:51:06 +02:00
Mustafa Paltun 457697f039 Update t_stream.c 2018-06-27 16:35:30 +03:00
Mustafa Paltun 47515fa348 Update help.h 2018-06-27 16:33:31 +03:00
Oran Agra 9df4b2b9b4 minor fix in creating a stream NACK for rdb and defrag tests 2018-06-27 15:34:17 +03:00
antirez 8672ffbe0b CLIENT UNBLOCK implemented. 2018-06-27 14:08:42 +02:00
antirez 97842a346d Take clients in a ID -> Client handle dictionary. 2018-06-27 14:08:42 +02:00
antirez c31ae4f8f4 CLIENT ID implemented. 2018-06-27 14:08:42 +02:00
Salvatore Sanfilippo ddbb355d96 Merge pull request #5063 from oranagra/test_suite_improvements
test suite infra improvements and fix
2018-06-27 14:07:25 +02:00
Salvatore Sanfilippo 91166f54fd Merge pull request #5065 from oranagra/defrag_jemalloc5
defrag hint support for jemalloc 5, and active defrag for streams
2018-06-27 14:05:16 +02:00
Oran Agra cfa076f2ee add active defrag support for streams 2018-06-27 15:00:41 +03:00
Oran Agra c4f4a2c3ef add defrag hint support into jemalloc 5 2018-06-27 10:52:39 +03:00
Oran Agra 9cb031adaa test suite infra improvements and fix
* fail the test (exit code) in case of timeout.
* add --wait-server to allow attaching a debugger
* add --dont-clean to keep log files when tests are done
2018-06-26 20:23:55 +03:00
Salvatore Sanfilippo 22b87358a2 Merge pull request #5027 from tigertv/unstable
Update sort.c
2018-06-26 16:13:20 +02:00
antirez 7f1573dacd Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-26 16:09:09 +02:00
antirez 30b75cef3b Sentinel: test command renaming feature. 2018-06-26 16:08:43 +02:00
Salvatore Sanfilippo 9b071acc7c Merge pull request #5059 from guybe7/fix_restore_warning
Fix compiler warning in restoreCommand
2018-06-26 14:41:48 +02:00
antirez 9b15b5cde3 Fix type of argslen in sendSynchronousCommand().
Related to #5037.
2018-06-26 14:38:35 +02:00
antirez 2a24e4dbeb Remove black space. 2018-06-26 14:37:22 +02:00
Salvatore Sanfilippo b5cd7336d9 Merge pull request #5037 from madolson/repl-auth-fix
Fixed replication authentication with whitespace password
2018-06-26 14:33:20 +02:00
antirez db2671e28b Sentinel: fix SENTINEL SET error reporting.
Thanks to @shenlongxing for reporting the problem.
Related to #5062.
2018-06-26 09:17:38 +02:00
Madelyn Olson 0fe8d9889b Addressed comments 2018-06-26 00:57:35 +00:00
Madelyn Olson d0c04bd6f5 Fixed replication authentication with whitespace in password 2018-06-26 00:48:37 +00:00
antirez 95612d5db3 Sentinel: drop the renamed-command entry in a more natural way.
Instead of telling the user to set the renamed command to "" to remove
the renaming, to the obvious thing when a command is renamed to itself.

So if I want to remove the renaming of PING, I just rename it to PING
again.
2018-06-25 17:50:46 +02:00
antirez 8b169ba4b7 Sentinel command renaming: document it into sentinel.conf. 2018-06-25 17:45:01 +02:00
antirez 9604d86756 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-25 17:34:12 +02:00
antirez 7c1c878e2c Sentinel command renaming: use case sensitive hashing for the dict. 2018-06-25 17:31:57 +02:00
antirez 80ec775b61 Sentinel command renaming: fix CONFIG SET event logging. 2018-06-25 17:24:04 +02:00
antirez 97f75f1589 Sentinel command renaming: fix CONFIG SET after refactoring. 2018-06-25 17:23:32 +02:00
antirez 5ae54e42b6 Sentinel command renaming: implement SENTINEL SET. 2018-06-25 17:13:20 +02:00
antirez bb4770af4b Sentinel: make SENTINEL SET able to handle different arities. 2018-06-25 17:12:39 +02:00
antirez 38e42968fa Sentinel command renaming: config rewriting. 2018-06-25 16:55:01 +02:00
antirez 6aad61d9ed Sentinel command renaming: rename-command option parsing. 2018-06-25 16:47:50 +02:00
antirez 52e72ee34b Sentinel command renaming: base machanism implemented. 2018-06-25 14:06:05 +02:00
Guy Benoish 0c92e3ec86 Fix compiler warning in restoreCommand 2018-06-24 16:53:01 +07:00
Salvatore Sanfilippo fc52e9216e Merge pull request #4634 from soloestoy/special-auth
config: handle special configuration "" for auth
2018-06-22 15:48:33 +02:00
Salvatore Sanfilippo 578583e31b Merge pull request #5049 from youjiali1995/fix-load-rdb
Fix rdbLoadIntegerObject() to create shared objects when needed.
2018-06-22 15:47:45 +02:00
Salvatore Sanfilippo 37c457c0c7 Merge pull request #5053 from michael-grunder/zpopminmax-keypos
Update ZPOPMIN/ZPOPMAX command declaration
2018-06-22 15:35:36 +02:00
Salvatore Sanfilippo f56006254e Merge pull request #5050 from shenlongxing/fix-typo
fix typo
2018-06-22 15:34:36 +02:00
michael-grunder 773c63eb49 Update ZPOPMIN/ZPOPMAX command declaration
Unlike the BZPOP variants, these functions take a single key.  This fixes
an erroneous CROSSSLOT error when passing a count to a cluster enabled
server.
2018-06-21 12:30:42 -07:00
Salvatore Sanfilippo c5f3e70cc3 Merge pull request #5051 from oranagra/streams_mem_estimate
fix streams memory estimation, missing raxSeek
2018-06-21 17:36:28 +02:00
Oran Agra d6fe68eaea fix streams memory estimation, missing raxSeek 2018-06-21 17:58:29 +03:00
shenlongxing bdb2664a81 fix typo 2018-06-21 22:08:09 +08:00
youjiali1995 12cd55a62e Fix rdbLoadIntegerObject() to create shared objects when needed. 2018-06-21 19:23:16 +08:00
Salvatore Sanfilippo 71b8520be6 Merge pull request #5036 from bepahol/unstable
Update README.md
2018-06-20 17:42:37 +02:00
Salvatore Sanfilippo f0d8e44f7c Merge pull request #5039 from oranagra/rdb_dbsize_hint
64 bit RDB_OPCODE_RESIZEDB in rdb saving
2018-06-20 11:18:31 +02:00
Salvatore Sanfilippo 35a6b461d7 Merge pull request #5040 from oranagra/memrev64ifbe_fix
use safe macro (non empty) in memrev64ifbe to eliminate empty if warning
2018-06-20 11:17:06 +02:00
Salvatore Sanfilippo 3e5624ac6b Merge pull request #5045 from guybe7/restore_fix
Enhance RESTORE with RDBv9 new features
2018-06-20 11:11:39 +02:00
Guy Benoish 56163a78a4 Enhance RESTORE with RDBv9 new features
RESTORE now supports:
1. Setting LRU/LFU
2. Absolute-time TTL

Other related changes:
1. RDB loading will not override LRU bits when RDB file
   does not contain the LRU opcode.
2. RDB loading will not set LRU/LFU bits if the server's
   maxmemory-policy does not match.
2018-06-20 15:11:08 +07:00
Salvatore Sanfilippo 7eac86ef6d Merge pull request #5042 from oranagra/malloc_usable_size_libc
add malloc_usable_size for libc malloc
2018-06-19 17:22:36 +02:00
Oran Agra 505956c6f9 add malloc_usable_size for libc malloc
this reduces the extra 8 bytes we save before each pointer.
but more importantly maybe, it makes the valgrind runs to be more similiar
to our normal runs.

note: the change in malloc_stats struct in server.h is to eliminate an name conflict.
structs that are not typedefed are resolved from a separate name space.
2018-06-19 18:18:23 +03:00
Salvatore Sanfilippo 65a79bc548 Merge pull request #5023 from FX-HAO/unstable
Fix update_zmalloc_stat_alloc in zrealloc
2018-06-19 16:50:22 +02:00
antirez ab27d9b08f Test RDB stream encoding saving/loading. 2018-06-19 16:29:15 +02:00
Salvatore Sanfilippo 3d34368dbe Merge pull request #5041 from oranagra/redis-rdb-check_rdbLoadMillisecondTime
fix redis-rdb-check to provide proper arguments to rdbLoadMillisecondTime
2018-06-19 16:06:11 +02:00
antirez 88c5e317c6 Modules: convert hash to hash table for big objects. 2018-06-19 16:03:00 +02:00
Oran Agra 56cbc978fc fix redis-rdb-check to provide proper arguments to rdbLoadMillisecondTime
due to incorrect forward declaration, it didn't provide all arguments.
this lead to random value being read from the stack and return of incorrect time,
which in this case doesn't matter since no one uses it.
2018-06-19 16:54:22 +03:00
antirez 7d0b46ec00 AOF: remove no longer used variable "now". 2018-06-19 15:54:05 +02:00
antirez b935ba95eb Modify clusterRedirectClient() to handle ZPOP and XREAD. 2018-06-19 15:53:32 +02:00
Oran Agra d575eae926 use safe macro (non empty) in memrev64ifbe to eliminate empty if warning 2018-06-19 16:46:41 +03:00
Oran Agra 5dc8ef5b47 64 bit RDB_OPCODE_RESIZEDB in rdb saving
this complication in the code is from times were rdbSaveLen didn't support 64 bits.
2018-06-19 16:43:12 +03:00
antirez bd094a4d5b Remove AOF optimization to skip expired keys.
Basically we cannot be sure that if the key is expired while writing the
AOF, the main thread will surely find the key expired. There are
possible race conditions like the moment at which the "now" is sampled,
and the fact that time may jump backward.

Think about the following:

SET a 5
EXPIRE a 1

AOF rewrite starts after about 1 second. The child process finds the key
expired, while in the main thread instead an INCR command is called
against the key "a" immediately after a fork, and the scheduler was
faster to give execution time to the main thread, so "a" is yet not
expired.

The main thread will generate an INCR a command to the AOF log that will
be appended to the rewritten AOF file, but that INCR command will target
a non existin "a" key, so a new non volatile key "a" will be created.

Two observations:

A) In theory by computing "now" before the fork, we should be sure that
if a key is expired at that time, it will be expired later when the
main thread will try to access to such key. However this does not take
into account the fact that the computer time may jump backward.

B) Technically we may still make the process safe by using a monotonic
time source.

However there were other similar related bugs, and in general the new
"vision" is that Redis persistence files should represent the memory
state without trying to be too smart: this makes the design more
consistent, bugs less likely to arise from complex interactions, and in
the end what is to fix is the Redis expire process to have less expired
keys in RAM.

Thanks to Oran Agra and Guy Benoish for writing me an email outlining
this problem, after they conducted a Redis 5 code review.
2018-06-19 15:43:06 +02:00
Benjamin Holst bdf798068e Update README.md 2018-06-18 22:41:26 -07:00
antirez b7559da532 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-18 17:09:16 +02:00
antirez 84d2884b72 Revert fix #4976 just leaving the flush() part. 2018-06-18 17:09:00 +02:00
antirez 68503f2af2 Fix incrDecrCommand() to create shared objects when needed.
See #5011.
2018-06-18 16:56:31 +02:00
antirez 97c0ca462d Refactor createObjectFromLongLong() to be suitable for value objects. 2018-06-18 16:55:16 +02:00
Salvatore Sanfilippo 222d874de4 Merge pull request #5020 from shenlongxing/fix-config
Fix config_set_numerical_field() integer overflow.
2018-06-18 16:02:23 +02:00
antirez c215f94a2c Streams: fix xreadGetKeys() for correctness.
The old version could not handle the fact that "STREAMS" is a valid key
name for streams. Now we really try to parse the command like the
command implementation would do.

Related to #5028 and 4857.
2018-06-18 14:06:06 +02:00
Salvatore Sanfilippo e61571f3bd Merge pull request #4857 from youjiali1995/fix-command-getkeys
Fix core dump when using some commands with wrong arguments.
2018-06-18 13:54:38 +02:00
antirez 6ec405c989 Streams: fix xreadGetKeys() buffer overflow.
The loop allocated a buffer for the right number of keys positions, then
overflowed it going past the limit.

Related to #4857 and cause of the memory violation seen in #5028.
2018-06-18 13:51:19 +02:00
antirez a260633ad6 Streams: Change XADD MAXLEN handling of values <= 0.
Now a MAXLEN of 0 really does what it means: it will create a zero
entries stream. This is useful in order to make sure that the behavior
is identical to XTRIM, that must be able to reduce the stream to zero
elements when MAXLEN is given.

Also now MAXLEN with a count < 0 will return an error.
2018-06-18 10:05:21 +02:00
Max Vetrov 8bc23f5d28 Update sort.c
Redundant second if, and we may use "not" operation for bool, I suppose it's faster.
2018-06-17 21:25:51 +02:00
antirez a6fc4dfcc6 XADD MAXLEN should return an error for values < 0. 2018-06-17 10:44:01 +02:00
Salvatore Sanfilippo 68d6acdd04 Merge pull request #4976 from trevor211/fixDebugLoadaof
Critical: Fix server crash and data inconsistency in some cases.
2018-06-16 11:05:04 +02:00
Salvatore Sanfilippo a5c30009e3 Merge pull request #4758 from soloestoy/rdb-save-incremental-fsync
Rdb save incremental fsync
2018-06-16 10:59:37 +02:00
antirez 731705eee1 Sentinel: add an option to deny online script reconfiguration.
The ability of "SENTINEL SET" to change the reconfiguration script at
runtime is a problem even in the security model of Redis: any client
inside the network may set any executable to be ran once a failover is
triggered.

This option adds protection for this problem: by default the two
SENTINEL SET subcommands modifying scripts paths are denied. However the
user is still able to rever that using the Sentinel configuration file
in order to allow such a feature.
2018-06-14 18:57:58 +02:00
antirez c2dbda9b6a Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-14 18:11:04 +02:00
antirez 66b99d9639 Fix infinite loop in dbRandomKey().
Thanks to @kevinmcgehee for signaling the issue and reasoning about the
consequences and potential fixes.

Issue #5015.
2018-06-14 18:08:21 +02:00
Salvatore Sanfilippo 162be7f88c Merge pull request #5008 from zwkno1/unstable
fix geohashEncode bug when step > 30
2018-06-14 14:01:40 +02:00
Salvatore Sanfilippo f2bd8c74dd Merge pull request #5021 from soloestoy/fix-exists
fix exists command on slave
2018-06-14 13:58:30 +02:00
Salvatore Sanfilippo 705d27d642 Merge pull request #5018 from soloestoy/optimize-reply
optimize reply list memory usage
2018-06-14 13:32:00 +02:00
antirez 5bcfe3dc38 Fix SCAN bug regression test, avoiding empty SREM call. 2018-06-14 12:21:58 +02:00
Fuxin Hao a979061f86 Fix update_zmalloc_stat_alloc in zrealloc 2018-06-14 16:44:19 +08:00
zhaozhao.zz d836647376 fix exists command on slave 2018-06-14 01:30:07 +08:00
shenlongxing 7a86e8704d Fix config_set_numerical_field() integer overflow. 2018-06-13 23:27:53 +08:00
zhaozhao.zz 1443ce8c38 optimize reply list memory usage 2018-06-13 20:35:40 +08:00
antirez 47f3617541 Security: fix redis-cli buffer overflow.
Thanks to Fakhri Zulkifli for reporting it.

The fix switched to dynamic allocation, copying the final prompt in the
static buffer only at the end.
2018-06-13 12:40:33 +02:00
antirez c622a8abf8 Security: fix Lua struct package offset handling.
After the first fix to the struct package I found another similar
problem, which is fixed by this patch. It could be reproduced easily by
running the following script:

    return struct.unpack('f', "xxxxxxxxxxxxx",-3)

The above will access bytes before the 'data' pointer.
2018-06-13 12:40:33 +02:00
antirez 22e656c9f3 Security: more cmsgpack fixes by @soloestoy.
@soloestoy sent me this additional fixes, after searching for similar
problems to the one reported in mp_pack(). I'm committing the changes
because it was not possible during to make a public PR to protect Redis
users and give Redis providers some time to patch their systems.
2018-06-13 12:40:33 +02:00
antirez 26fb954458 Security: update Lua struct package for security.
During an auditing Apple found that the "struct" Lua package
we ship with Redis (http://www.inf.puc-rio.br/~roberto/struct/) contains
a security problem. A bound-checking statement fails because of integer
overflow. The bug exists since we initially integrated this package with
Lua, when scripting was introduced, so every version of Redis with
EVAL/EVALSHA capabilities exposed is affected.

Instead of just fixing the bug, the library was updated to the latest
version shipped by the author.
2018-06-13 12:40:33 +02:00
antirez 3c1c589a24 Security: fix Lua cmsgpack library stack overflow.
During an auditing effort, the Apple Vulnerability Research team discovered
a critical Redis security issue affecting the Lua scripting part of Redis.

-- Description of the problem

Several years ago I merged a pull request including many small changes at
the Lua MsgPack library (that originally I authored myself). The Pull
Request entered Redis in commit 90b6337c1, in 2014.
Unfortunately one of the changes included a variadic Lua function that
lacked the check for the available Lua C stack. As a result, calling the
"pack" MsgPack library function with a large number of arguments, results
into pushing into the Lua C stack a number of new values proportional to
the number of arguments the function was called with. The pushed values,
moreover, are controlled by untrusted user input.

This in turn causes stack smashing which we believe to be exploitable,
while not very deterministic, but it is likely that an exploit could be
created targeting specific versions of Redis executables. However at its
minimum the issue results in a DoS, crashing the Redis server.

-- Versions affected

Versions greater or equal to Redis 2.8.18 are affected.

-- Reproducing

Reproduce with this (based on the original reproduction script by
Apple security team):

https://gist.github.com/antirez/82445fcbea6d9b19f97014cc6cc79f8a

-- Verification of the fix

The fix was tested in the following way:

1) I checked that the problem is no longer observable running the trigger.
2) The Lua code was analyzed to understand the stack semantics, and that
actually enough stack is allocated in all the cases of mp_pack() calls.
3) The mp_pack() function was modified in order to show exactly what items
in the stack were being set, to make sure that there is no silent overflow
even after the fix.

-- Credits

Thank you to the Apple team and to the other persons that helped me
checking the patch and coordinating this communication.
2018-06-13 12:40:33 +02:00
antirez 6dbc7987d7 RDB: Apply fix to rdbLoadMillisecondTime() only for new RDB versions.
This way we let big endian systems to still load old RDB versions.
However newver versions will be saved and loaded in a way that make RDB
expires cross-endian again. Thanks to @oranagra for the reporting and
the discussion about this problem, leading to this fix.
2018-06-12 18:21:39 +02:00
antirez 2ec4d9be58 Streams: generate a few additional events.
Currently it does not look it's sensible to generate events for streams
consumer groups modification, being them metadata, however at least for
key-level events, like the creation or removal of a consumer group, I
added a few events here and there. Later we can evaluate if it makes
sense to add more. From the POV instead of WAIT (in Redis transaciton)
and signaling the key as modified, it looks like that the transaction
should not fail when a stream is modified, so no calls are made in
consumer groups related functions to signalModifiedKey().
2018-06-12 18:11:15 +02:00
antirez cfeffb1afe Fix rdbSaveKeyValuePair() integer overflow.
Again thanks to @oranagra. The object idle time does not fit into an int
sometimes: use the native type that the serialization function will get
as argument, which is uint64_t.
2018-06-12 17:31:04 +02:00
antirez d9a2f80281 In scanDatabaseForReadyLists() now we need to handle ZSETs as well.
Since the introduction of ZPOP makes this needed. Thanks to @oranagra
for reporting.
2018-06-12 17:28:40 +02:00
antirez 49afd7d4c5 RDB: store times consistently in little endian.
I'm not sure how this escaped the attention of Redis users for years,
but finally @oranagra reported this issue... Thanks to Oran.
2018-06-12 17:22:03 +02:00
antirez 048815f32e Streams: improve type correctness in t_stream.c. 2018-06-12 14:12:53 +02:00
Itamar Haber 2dbca202b7 Applies addReplySubSyntaxError to stream commands 2018-06-12 14:52:50 +03:00
antirez 41824b9b20 Fix XGROUP help missing space. 2018-06-12 13:20:46 +02:00
antirez ca9902cec7 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-12 13:14:01 +02:00
antirez ec4093dc4d Regression test for issue #5006. 2018-06-12 13:13:35 +02:00
Salvatore Sanfilippo 78ae4d647c Merge pull request #5012 from shenlongxing/fix-config
fix active-defrag-threshold value error
2018-06-12 13:05:19 +02:00
Shen Longxing 67d2e3d65b fix active-defrag-threshold value error
The active-defrag-threshold-lower/active-defrag-threshold-upper min/max  value in redis.conf should be consistent with 'config set' command.
2018-06-12 17:59:32 +08:00
antirez 77c72d5e40 Streams: fix backward iteration when entry is not flagged SAMEFIELD.
See issue #5006. The comment in the code was also wrong and
was rectified as well.
2018-06-12 10:22:49 +02:00
球状闪电 a882df74e0 Update geohash.c
fix geohasEncode bug when step > 31
2018-06-12 15:28:28 +08:00
Salvatore Sanfilippo 1e15565b37 Merge pull request #5007 from leonchen83/patch-2
fix typo issue #5005
2018-06-12 09:28:26 +02:00
antirez 2e9812bf9c Streams: increment dirty counter for XGROUP SETID/DESTROY.
See issue #5005 comments.
2018-06-12 09:27:40 +02:00
Baoyi Chen 200f4e28ad fix typo
fix [#5005](https://github.com/antirez/redis/issues/5005)
2018-06-12 08:52:18 +08:00
antirez 159347cef1 Use a less aggressive query buffer resize policy.
A user with many connections (10 thousand) on a single Redis server
reports in issue #4983 that sometimes Redis is idle becuase at the same
time many clients need to resize their query buffer according to the old
policy.

It looks like this was created by the fact that we allow the query
buffer to grow without problems to a size up to PROTO_MBULK_BIG_ARG
normally, but when the client is idle we immediately are more strict,
and a query buffer greater than 1024 bytes is already enough to trigger
the resize. So for instance if most of the clients stop at the same time
this issue should be easily triggered.

This behavior actually looks odd, and there should be only a clear limit
after we say, let's look at this query buffer to check if it's time to
resize it. This commit puts the limit at PROTO_MBULK_BIG_ARG, and the
check is performed both if compared to the peak usage the current usage
is too big, or if the client is idle.

Then when the check is performed, to waste just a few kbytes is
considered enough to proceed with the resize. This should fix the issue.
2018-06-11 17:12:28 +02:00
antirez 10c4c45137 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-11 16:52:45 +02:00
antirez e2f96f3eab Fix client unblocking for XREADGROUP, issue #4978.
We unblocked the client too early, when the group name object was no
longer valid in client->bpop, so propagating XCLAIM later in
streamPropagateXCLAIM() deferenced a field already set to NULL.
2018-06-11 16:51:06 +02:00
Salvatore Sanfilippo d762172729 Merge pull request #5000 from shenlongxing/fix-config
fix integer case error
2018-06-11 16:34:26 +02:00
Salvatore Sanfilippo 17a42cc769 Merge pull request #4995 from shenlongxing/unstable
fix stream config typo
2018-06-11 16:33:38 +02:00
Salvatore Sanfilippo b578ab0bf4 Merge pull request #5002 from soloestoy/streams-read-or-write
Streams: lookupKey[Read->Write]OrReply in xdel and xtrim
2018-06-11 16:33:10 +02:00
Salvatore Sanfilippo f49b7e0d1d Merge pull request #4901 from KFilipek/zmalloc_typo_fix
HW_PHYSMEM typo in preprocessor condition
2018-06-11 16:32:40 +02:00
Salvatore Sanfilippo e54b49ac4c Merge pull request #5003 from soloestoy/streams-checkType
Streams: checkType for xread & xinfo
2018-06-11 16:32:00 +02:00
Salvatore Sanfilippo 73fef1b0b8 Merge pull request #5001 from michael-grunder/xgroup-check-type
Abort in XGROUP if the key is not a stream
2018-06-11 16:30:49 +02:00
antirez a78b99890e Improved regression test for #4906.
Removing the fix about 50% of the times the test will not be able to
pass cleanly. It's very hard to write a test that will always fail, or
actually, it is possible but then it's likely that it will consistently
pass if we change some random bit, so better to use randomization here.
2018-06-11 13:10:06 +02:00
antirez 3161c7248d Regression test for the dictScan() issue #4906. 2018-06-11 12:51:26 +02:00
zhaozhao.zz 83bb4a0dff Streams: checkType for xread & xinfo 2018-06-11 01:17:52 +08:00
zhaozhao.zz b47471880d Streams: lookupKey[Read->Write]OrReply in xdel and xtrim 2018-06-11 00:14:17 +08:00
michael-grunder efcc676b0b Abort in XGROUP if the key is not a stream 2018-06-10 08:55:58 -07:00
Itamar Haber 8a2eca39ce Merge branch 'unstable' into module_command_help 2018-06-09 21:10:53 +03:00
Itamar Haber 4ef817295e Capitalizes subcommands & orders lexicographically 2018-06-09 21:03:52 +03:00
Itamar Haber 4502f10789 Capitalizes subscommands 2018-06-09 20:54:05 +03:00
shenlongxing 6e6c13123e fix integer case error 2018-06-09 22:24:33 +08:00
Salvatore Sanfilippo b7df488618 Merge pull request #4519 from soloestoy/zset-int-problem
Zset int problem
2018-06-08 12:45:11 +02:00
Salvatore Sanfilippo 980969ec59 Merge pull request #4800 from soloestoy/dict-expand
adjust position of _dictNextPower in dictExpand
2018-06-08 12:32:18 +02:00
Salvatore Sanfilippo 5ff23db284 Merge pull request #4861 from soloestoy/rdb-dict-expand
RDB: expand dict if needed when rdb load object
2018-06-08 12:12:34 +02:00
antirez d2c353280e Implement DEBUG htstats-key. 2018-06-08 11:17:20 +02:00
antirez 08da518c19 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-08 11:02:30 +02:00
antirez 52e4b9eda3 redis-cli inline help updated. 2018-06-07 18:53:00 +02:00
antirez 8fe34cec06 Add the stream group to the script generating the help. 2018-06-07 18:52:01 +02:00
Salvatore Sanfilippo ac0b8125e0 Merge pull request #4977 from michael-grunder/xpending-early-return
Return early in XPENDING if sent a nonexistent consumer group.
2018-06-07 18:49:09 +02:00
Itamar Haber 594dc58322 Globally applies addReplySubSyntaxError 2018-06-07 18:39:36 +03:00
Itamar Haber 041354e665 Adds MODULE HELP and implements addReplySubSyntaxError 2018-06-07 18:34:58 +03:00
shenlongxing bce0072026 fix stream config typo 2018-06-07 23:13:36 +08:00
antirez 10a88d5c86 Streams: better document the max node limits. 2018-06-07 14:37:56 +02:00
antirez 664cfb8d6b Typo: entires -> entries in several places. 2018-06-07 14:36:56 +02:00
antirez 5c7bda3752 Streams: make macro node limits configurable. 2018-06-07 14:36:13 +02:00
antirez 80f89e0634 Streams: max node limits only checked if non zero. 2018-06-07 14:30:25 +02:00
antirez 89435af570 Streams: use non static macro node limits.
Also add the concept of size/items limit, instead of just having as
limit the number of bytes.
2018-06-07 14:24:49 +02:00
shenlongxing c6c03ee21f Fix write() errno error 2018-06-06 13:06:42 +02:00
antirez 3406400ae0 Fix streamIteratorRemoveEntry() to update elements count.
Close #4989.
2018-06-06 11:40:22 +02:00
antirez c8d43b84d8 ZPOP: invert score-ele to match ZRANGE WITHSCORES order. 2018-06-05 17:06:25 +02:00
antirez d3b425470b Remove XINFO <key> special form.
As observed by Michael Grunder this usage while practical is
inconsistent because for instance it does not work against a key called
HELP. Removed.
2018-06-05 16:34:34 +02:00
antirez 7932b157c2 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-06-04 17:30:11 +02:00
antirez ff7cb41475 XGROUP SETID implemented + consumer groups core fixes.
Now that we have SETID, the inetrnals of consumer groups should be able
to handle the case of the same message delivered multiple times just
as a side effect of calling XREADGROUP. Normally this should never
happen but if the admin manually "XGROUP SETID mykey mygroup 0",
messages will get re-delivered to clients waiting for the ">" special
ID. The consumer groups internals were not able to handle the case of a
message re-delivered in this circumstances that was already assigned to
another owner.
2018-06-04 17:28:03 +02:00
antirez 0f1448f9c3 Rax library updated. 2018-06-04 17:26:16 +02:00
Salvatore Sanfilippo 45ab995a3a Merge pull request #4981 from yossigo/gcc7
Clean gcc 7.x warnings, redis-cli cluster fix.
2018-06-04 17:18:39 +02:00
antirez 226712b467 XGROUP DESTROY implemented. 2018-06-04 12:58:26 +02:00
赵磊 366c45a2ec Fix core dump when using 'command getkeys' with wrong arguments. 2018-06-04 15:14:50 +08:00
Yossi Gottlieb 7c24f28f24 Clean gcc 7.x warnings, redis-cli cluster fix. 2018-06-03 15:54:30 +03:00
michael-grunder ac6bb9ff05 Return early in XPENDING if sent a nonexistent consumer group. 2018-06-02 18:22:20 -07:00
WuYunlong f196b77478 Fix DEBUG LOADAOF so that redis-server will not crash unexpectedly
and will not be inconsistent after we call debug loadaof.
Before this commit, there were 2 problems:

1, When appendonly is set to no and there is not a appendonly file,
   redis-server will crash if we call DEBUG LOADAOF.
2, When appendonly is set to no and there is a appendonly file,
   redis-server will hold different data after loading  appendonly
   file.
2018-06-02 14:32:16 +08:00
Salvatore Sanfilippo 5418bfb8e8 Merge pull request #4907 from youjiali1995/fix-dictScan
Fix dictScan(): It can't scan all buckets when dict is shrinking.
2018-06-01 16:54:05 +02:00
artix f5dad061ee Cluster Manager: fixed master_id check in clusterManagerNodeLoadInfo 2018-06-01 16:19:47 +02:00
Salvatore Sanfilippo 252cf5e319 Merge pull request #4971 from soloestoy/bugfix-bzpop
ZPOP: unblock multiple clients in right way
2018-05-31 18:40:53 +02:00
zhaozhao.zz f0e81aff07 ZPOP: unblock multiple clients in right way 2018-05-31 23:35:47 +08:00
Salvatore Sanfilippo d4c28bc485 Merge pull request #4945 from GitHubMota/debug_recover
Fix debug crash-and-recover help info.
2018-05-31 17:32:11 +02:00
Salvatore Sanfilippo 927b581838 Merge pull request #4953 from soloestoy/fix-memory-overhead
MEMORY: fix the missing of monitor clients buffers
2018-05-31 17:31:24 +02:00
Salvatore Sanfilippo 75fa316cfd Merge pull request #4964 from remicollet/issue-uint64
include stdint.h for unit64_t definition
2018-05-31 17:20:33 +02:00
antirez c513733526 Capitalize OBJECT HELP subcommands. 2018-05-31 17:11:46 +02:00
artix 4edb920536 Cluster Manager: fixed infinite loop in rebalance (Issue #4941). 2018-05-31 15:59:16 +02:00
Remi Collet 6c4c2a2432 include stdint.h for unit64_t definition 2018-05-30 15:33:06 +02:00
antirez 71b745b612 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-05-29 12:41:07 +02:00
antirez 9e81213746 Don't expire keys while loading RDB from AOF preamble.
The AOF tail of a combined RDB+AOF is based on the premise of applying
the AOF commands to the exact state that there was in the server while
the RDB was persisted. By expiring keys while loading the RDB file, we
change the state, so applying the AOF tail later may change the state.

Test case:

* Time1: SET a 10
* Time2: EXPIREAT a $time5
* Time3: INCR a
* Time4: PERSIT A. Start bgrewiteaof with RDB preamble. The value of a is 11 without expire time.
* Time5: Restart redis from the RDB+AOF: consistency violation.

Thanks to @soloestoy for providing the patch.
Thanks to @trevor211 for the original issue report and the initial fix.

Check issue #4950 for more info.
2018-05-29 12:37:42 +02:00
WuYunlong 09ae1a46d6 Fix rdb save by allowing dumping of expire keys, so that when
we add a new slave, and do a failover, eighter by manual or
not, other local slaves will delete the expired keys properly.
2018-05-29 12:35:15 +02:00
zhaozhao.zz 8840ab4bd7 MEMORY: fix the missing of monitor clients buffers 2018-05-27 16:34:58 +08:00
antirez 63bbacc4ab Streams: fix comments referring to old ID format. 2018-05-25 17:25:19 +02:00
Salvatore Sanfilippo c2d7c5bbcd Merge pull request #4948 from soloestoy/avoid-warning-stream
avoid warning in t_stream.c
2018-05-25 17:00:46 +02:00
antirez dd4ac1654b Streams: fix test ID format. 2018-05-25 16:57:08 +02:00
zhaozhao.zz 3b2b5e1d1b avoid warning in t_stream.c 2018-05-25 22:49:49 +08:00
antirez a17226d419 Backport hiredis issue 525 fix to compile on FreeBSD.
Close #4947.
2018-05-25 16:33:23 +02:00
antirez 1ea37fcf4f Simplify XADD ID syntax error code path,. 2018-05-25 16:29:37 +02:00
Salvatore Sanfilippo a998d21f71 Merge pull request #4832 from 0xtonyxia/fix-xadd-ID-parse
Bugfix: xadd command ID parse
2018-05-25 16:27:21 +02:00
antirez 8bdaeaa4c1 Avoid warning by explicitly initializing var to NULL. 2018-05-25 16:19:14 +02:00
antirez a881942e60 Explicitly cast void pointers to char pointers for printf style logging. 2018-05-25 16:15:59 +02:00
Salvatore Sanfilippo 9f2abcd020 Merge pull request #4946 from soloestoy/linux-make-lrt
append -lrt to FINAL_LIBS for linux
2018-05-25 16:12:19 +02:00
zhaozhao.zz 4e8a00e6b0 append -lrt to FINAL_LIBS for linux 2018-05-25 20:16:57 +08:00
antirez ad984d169b Don't append libjemalloc.a otherwise it will be after -ldl. 2018-05-25 13:36:51 +02:00
Mota c9a10a9419 Fix debug crash-and-recover help info. 2018-05-25 16:49:23 +08:00
antirez 2fa611607b Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-05-24 18:21:23 +02:00
antirez 60ead966fb Generate configure for Jemalloc. 2018-05-24 18:21:13 +02:00
antirez f3db15d2c9 Make active defragmentation tests optional.
They failed when active defrag could not be activated because the
Jemalloc version does not include the additional APIs.
2018-05-24 18:04:21 +02:00
antirez 3e071d1f9a Add --with-version in Jemalloc config script. 2018-05-24 17:50:12 +02:00
antirez 1af1d38775 Specify that Jemalloc configure requires version. 2018-05-24 17:49:36 +02:00
antirez 3c098ab96f Jemalloc upgraded to version 5.0.1. 2018-05-24 17:17:37 +02:00
antirez 6fe1cbaa94 Update documentation about how to upgrade Jemalloc. 2018-05-24 17:16:12 +02:00
Salvatore Sanfilippo 13f646e8c4 Merge pull request #4099 from JunHe77/aarch64_stack_cflag
Fixed stack trace generation on aarch64
2018-05-24 16:16:12 +02:00
Salvatore Sanfilippo 3e493b1bca Merge pull request #4932 from 0xtonyxia/fix-redis-cli-set-prefer-memory-leak
Fix redis-cli memory leak when sending set :preference command.
2018-05-24 16:15:21 +02:00
Salvatore Sanfilippo 61f7f49070 Merge pull request #4922 from 0xtonyxia/enhance-cli-security
Enhance security for redis-cli
2018-05-24 16:14:42 +02:00
Salvatore Sanfilippo a99b41c816 Merge pull request #4831 from 0xtonyxia/fix-xadd-arity-check
Bugfix: xadd comand <field,value> arity check
2018-05-24 16:07:19 +02:00
antirez 2afb128ebc Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-05-23 17:14:59 +02:00
antirez d816a658e9 Sentinel: fix delay in detecting ODOWN.
See issue #2819 for details. The gist is that when we want to send INFO
because we are over the time, we used to send only INFO commands, no
longer sending PING commands. However if a master fails exactly when we
are about to send an INFO command, the PING times will result zero
because the PONG reply was already received, and we'll fail to send more
PINGs, since we try only to send INFO commands: the failure detector
will delay until the connection is closed and re-opened for "long
timeout".

This commit changes the logic so that we can send the three kind of
messages regardless of the fact we sent another one already in the same
code path. It could happen that we go over the message limit for the
link by a few messages, but this is not significant. However now we'll
not introduce delays in sending commands just because there was
something else to send at the same time.
2018-05-23 17:13:44 +02:00
Salvatore Sanfilippo 7f225a69a7 Merge pull request #4908 from soloestoy/aof-rdb-preamble-compatible-checksum-no
AOF & RDB: be compatible with rdbchecksum no
2018-05-23 17:11:00 +02:00
Salvatore Sanfilippo 0a726e9021 Merge pull request #4935 from soloestoy/bugfix-zpop
bugfix for ZPOP: fix the wrong keyc, should be 1
2018-05-22 15:52:21 +02:00
zhaozhao.zz 980f0e9d5e ZPOP: fix the wrong keyc, should be 1 2018-05-22 21:31:22 +08:00
dejun.xdj aac27d834d Fix redis-cli memory leak when sending set preference command. 2018-05-21 12:19:37 +08:00
dejun.xdj 5e0261c0eb Check if the repeat value is positive in while loop of cliSendCommand().
In case that the incoming repeat parameter is negative and causes a
deadless loop.
2018-05-21 12:06:48 +08:00
dejun.xdj fab8975463 Change the type of repeat argument to long for function cliSendCommand.
To be in consistent with the original definition.
2018-05-21 12:04:53 +08:00
dejun.xdj dc4be43b57 Fix negtive repeat command value issue.
If command like "-1 set a b" is sent with redis-cli, it will cause a deadless loop. So some repeat value checking logic is added to avoid this.
2018-05-19 22:50:40 +08:00
dejun.xdj 8ce9da0e9c Detect and stop saving history for auth command with repeat option.
Put the repeat option checking code a little forward to avoid repeat logic.
2018-05-18 11:40:05 +08:00
dejun.xdj 9eb550dc82 Change the warning message a little bit to avoid trademark issuses. 2018-05-18 11:37:31 +08:00
antirez 4c4d902075 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-05-17 16:53:14 +02:00
antirez 586598297b Merge remote-tracking branch 'artix/trib2cli' into unstable 2018-05-17 16:52:31 +02:00
Salvatore Sanfilippo eae6287d76 Merge pull request #4927 from oranagra/defrag_32bit_fix
Active defrag fixes for 32bit builds
2018-05-17 12:24:40 +02:00
Oran Agra 6ff807bd47 Active defrag fixes for 32bit builds
problems fixed:
* failing to read fragmentation information from jemalloc
* overflow in jemalloc fragmentation hint to the defragger
* test suite not triggering eviction after population
2018-05-17 09:52:00 +03:00
artix a41999e513 Removed TODO in redis-cli 2018-05-16 18:04:13 +02:00
artix 6b1be41055 Cluster Manager: fixed unprinted reply error 2018-05-16 17:49:18 +02:00
dejun.xdj 0e070f8e1e Stop saving auth command in redis-cli history. 2018-05-16 16:18:00 +08:00
dejun.xdj 50eeff67b0 Add warning message when using password on command line 2018-05-16 16:15:12 +08:00
artix d7fb3f1261 Cluster Manager: print flags as strings. 2018-05-15 18:41:46 +02:00
antirez 15001ba6c1 Test: fix blocking lists/zsets replication test.
By verifying that it was able to find a regression, and fixing it
accordingly.
2018-05-15 17:43:41 +02:00
antirez c42f0e1de0 Test: replication test blocking lists/zsets ops. 2018-05-15 17:33:29 +02:00
antirez 5e2dcccb71 ZPOP: fix replication of blocking ZPOP. 2018-05-15 16:03:56 +02:00
artix c84512a89e - Fixed mistyped redis command (clusterManagerGetNodeWithMostKeysInSlot)
- Cluster node structure is now updated after ADDSLOTS
2018-05-11 18:29:12 +02:00
antirez 44aa7f9257 ZPOP: fix the tests according to new non-blocking "count" argument. 2018-05-11 18:07:10 +02:00
antirez 897c8052ee ZPOP: change sync ZPOP to have a count argument instead of N keys.
Usually blocking operations make a lot of sense with multiple keys so
that we can listen to multiple queues (or whatever the app models) with
a single connection. However in the synchronous case it is more useful
to be able to ask for N elements. This is a change that I also wanted to
perform soon or later in the blocking list variant, but here it is more
natural since there is no reply type difference.
2018-05-11 18:00:32 +02:00
antirez a6f9c30ac4 ZPOP: renaming to have explicit MIN/MAX score idea.
This commit also adds a top comment about a subtle behavior of mixing
blocking operations of different types in the same key.
2018-05-11 17:31:53 +02:00
antirez a2cee38fd3 Merge branch 'zpop' of https://github.com/itamarhaber/redis into zpop 2018-05-11 10:40:13 +02:00
antirez f06c30d2f3 Fix rdb.c dictionary iterator release in 2 more places. 2018-05-09 12:06:37 +02:00
antirez 678f200b5b Fix rdb.c dictionary iterator release.
Some times it was not released on error, sometimes it was released two
times because the error path expected the "di" var to be NULL if the
iterator was already released. Thanks to @oranagra for pinging me about
potential problems of this kind inside rdb.c.
2018-05-09 11:03:27 +02:00
Salvatore Sanfilippo 98af8856a9 Merge pull request #4594 from smallstool/bugfix-freeMemoryIfneeded-int-overflow
fix int overflow problem in freeMemoryIfNeeded
2018-05-08 17:27:18 +02:00
zhaozhao.zz 3ab262e3da AOF & RDB: be compatible with rdbchecksum no 2018-05-08 19:22:13 +08:00
赵磊 cb48077b31 Fix dictScan(): It can't scan all buckets when dict is shrinking. 2018-05-08 15:30:11 +08:00
artix 7627338fb9 Cluster Manager: --cluster options can now be placed everywhere 2018-05-07 17:31:34 +02:00
artix ed6f173da4 - Updated create-cluster with redis-cli
- Updated README
2018-05-07 15:56:15 +02:00
Krzysztof Filipek b3c0868343 Typo in preprocessor condition 2018-05-06 20:18:48 +02:00
Itamar Haber 8048346e7d Adds memory information about the script's cache to INFO
Implementation notes: as INFO is "already broken", I didn't want to break it further. Instead of computing the server.lua_script dict size on every call, I'm keeping a running sum of the body's length and dict overheads.

This implementation is naive as it **does not** take into consideration dict rehashing, but that inaccuracy pays off in speed ;)

Demo time:

```bash
$ redis-cli info memory | grep "script"
used_memory_scripts:96
used_memory_scripts_human:96B
number_of_cached_scripts:0
$ redis-cli eval "" 0 ; redis-cli info memory | grep "script"
(nil)
used_memory_scripts:120
used_memory_scripts_human:120B
number_of_cached_scripts:1
$ redis-cli script flush ; redis-cli info memory | grep "script"
OK
used_memory_scripts:96
used_memory_scripts_human:96B
number_of_cached_scripts:0
$ redis-cli eval "return('Hello, Script Cache :)')" 0 ; redis-cli info memory | grep "script"
"Hello, Script Cache :)"
used_memory_scripts:152
used_memory_scripts_human:152B
number_of_cached_scripts:1
$ redis-cli eval "return redis.sha1hex(\"return('Hello, Script Cache :)')\")" 0 ; redis-cli info memory | grep "script"
"1be72729d43da5114929c1260a749073732dc822"
used_memory_scripts:232
used_memory_scripts_human:232B
number_of_cached_scripts:2
✔ 19:03:54 redis [lua_scripts-in-info-memory L ✚…⚑] $ redis-cli evalsha 1be72729d43da5114929c1260a749073732dc822 0
"Hello, Script Cache :)"
```
2018-04-30 19:33:01 +03:00
Itamar Haber e3e0a66adf Implements [B]Z[REV]POP and the respective unit tests
An implementation of the
[Ze POP Redis Module](https://github.com/itamarhaber/zpop) as core
Redis commands.

Fixes #1861.
2018-04-30 02:10:42 +03:00
artix fdabd59b7c Merge branch 'trib2cli' of github.com:artix75/redis into trib2cli 2018-04-23 16:29:29 +02:00
artix 2ba7fbfb94 Cluster tests now using redis-cli instead of redis-trib 2018-04-23 16:28:48 +02:00
artix 0886db7ba6 Cluster Manager: fixed expected slots calculation (rebalance)
Cluster Manager: fixed argument parsing after --cluster-weight
2018-04-23 16:28:48 +02:00
artix d7ae96327b Cluster Manager: fixed bug when parsing CLUSTER NODES reply (clusterManagerNodeLoadInfo) 2018-04-23 16:28:48 +02:00
artix be2f5c3ee3 Cluster Manager: code improvements and more comments added. 2018-04-23 16:28:48 +02:00
artix 08e41b49cf Cluster Manager: set-timeout command 2018-04-23 16:28:48 +02:00
artix 6ed5e32a5d - Cluster Manager: del-node command.
- Cluster Manager: fixed bug in clusterManagerNodeWithLeastReplicas
2018-04-23 16:28:48 +02:00
artix 3813fe12c4 Cluster Manager: add-node command. 2018-04-23 16:28:48 +02:00
artix d2d9fee4bc Cluster Manager: added clusterManagerCheckCluster to import command 2018-04-23 16:28:48 +02:00
artix 68f53b8be1 Cluster Manager: import command 2018-04-23 16:28:48 +02:00
artix 80f4553a37 Cluster Manager: fix command. 2018-04-23 16:28:48 +02:00
artix 4907218145 Cluster Manager: rebalance command 2018-04-23 16:28:48 +02:00
artix 98f67ebcfb clusterManagerAddSlots: changed the way ADDSLOTS command is built 2018-04-23 16:28:48 +02:00
artix e730b6e3ee ClusterManager: fixed --cluster-from 'all' parsing 2018-04-23 16:28:48 +02:00
Artix c5eca33917 Cluster Manager: fixed some memory error 2018-04-23 16:28:48 +02:00
artix 6e734eeedd Fixed memory write error in clusterManagerGetConfigSignature 2018-04-23 16:28:48 +02:00
artix 50d95cc5c8 Cluster Manager: reshard command, fixed slots
parsing bug and other minor bugs.
2018-04-23 16:28:48 +02:00
artix efe06fcecd Cluster Manager:
- Almost all Cluster Manager related code moved to
  the same section.
- Many macroes converted to functions
- Added various comments
- Little code restyling
2018-04-23 16:28:48 +02:00
artix a0b8992ea3 - Fixed bug in clusterManagerGetAntiAffinityScore
- Code improvements
2018-04-23 16:28:48 +02:00
artix b068d0fb13 Cluster Manager: colorized output 2018-04-23 16:28:48 +02:00
artix 3161a5bed0 Cluster Manager: improved cleanup/error handling in various functions 2018-04-23 16:28:48 +02:00
artix fa986e8cad Cluster Manager: 'call' command. 2018-04-23 16:28:48 +02:00
artix a0e1884b54 Cluster Manager: CLUSTER_MANAGER_NODE_CONNECT macro 2018-04-23 16:28:48 +02:00
artix a376881fc2 ClusterManager: added replicas count to clusterManagerNode 2018-04-23 16:28:48 +02:00
artix 653799a4d8 Cluster Manager: cluster is considered consistent if only one node has been found 2018-04-23 16:28:48 +02:00
artix 377717d6e1 Cluster Manager: reply error catch for MEET command 2018-04-23 16:28:48 +02:00
artix fda6c55176 Cluster Manager: slots coverage check. 2018-04-23 16:28:48 +02:00
artix 4ccca8e7a8 - Cluster Manager: fixed various memory leaks
- Cluster Manager: fixed flags assignment in
  clusterManagerNodeLoadInfo
2018-04-23 16:28:48 +02:00
artix d74107c05d Added check for open slots (clusterManagerCheckCluster) 2018-04-23 16:28:47 +02:00
artix be7458e4ce Cluster Manager: 'create', 'info' and 'check' commands 2018-04-23 16:28:47 +02:00
artix cf8d65a35d Cluster Manager mode 2018-04-23 16:28:47 +02:00
zhaozhao.zz c604f054b9 RDB: expand dict if needed when rdb load object 2018-04-22 22:30:44 +08:00
artix 363197ab7b Cluster tests now using redis-cli instead of redis-trib 2018-04-20 19:29:42 +02:00
artix 9e373c89b8 Cluster Manager: fixed expected slots calculation (rebalance)
Cluster Manager: fixed argument parsing after --cluster-weight
2018-04-20 19:25:10 +02:00
artix 4026c92b1b Cluster Manager: fixed bug when parsing CLUSTER NODES reply (clusterManagerNodeLoadInfo) 2018-04-20 18:08:30 +02:00
artix a3d5864e42 Cluster Manager: code improvements and more comments added. 2018-04-19 18:52:01 +02:00
antirez a968ee6fcd Streams: XTRIM command added. 2018-04-19 16:25:29 +02:00
antirez f918d0c0f7 Streams: XADD, handle signaling of write in different ways. 2018-04-19 16:11:43 +02:00
antirez 5bfd409b6e Streams: XDEL command. 2018-04-18 13:12:09 +02:00
antirez d13ff164bf Streams: higher level "delete by ID" API for streams. 2018-04-17 17:55:56 +02:00
antirez 852cc013ac Streams: iterator entry deletion abilities. 2018-04-17 17:18:00 +02:00
Salvatore Sanfilippo da87ffb3ae Merge pull request #4842 from soloestoy/fix-timer-api-wrong-insert
Modules Timer API: fix wrong raxInsert() usage
2018-04-13 19:51:46 +02:00
zhaozhao.zz ba2dbba0cb Modules Timer API: fix wrong raxInsert() usage 2018-04-14 01:20:02 +08:00
artix 7ace7af0b7 Cluster Manager: set-timeout command 2018-04-13 16:09:22 +02:00
antirez 7365cbf830 Modules API: Add call to get the blocked client handle from the context.
This is useful in the reply and timeout callback, if the module wants to
do some cleanup of the blocked client handle that may be stored around
in the module-private data structures.
2018-04-13 13:48:11 +02:00
antirez d7fdac673c Modules API: moduleGetReplyClient() refactoring. 2018-04-13 13:42:49 +02:00
antirez f305a7790d Modules API: blocked client disconnection callback. 2018-04-12 13:21:48 +02:00
antirez 3f9809d6fc Modules API: fix timer example. 2018-04-12 13:00:18 +02:00
artix 64cf7a314c - Cluster Manager: del-node command.
- Cluster Manager: fixed bug in clusterManagerNodeWithLeastReplicas
2018-04-11 18:23:28 +02:00
artix 5b9b7bb7ce Cluster Manager: add-node command. 2018-04-11 17:08:53 +02:00
antirez 643d5348cc Modules API: fix OOM_WARNING flag implementation. 2018-04-11 16:25:54 +02:00
antirez 936e29a1f3 Modules API: OOM_WARNING flags.
In some modules it may be useful to have an idea about being near to
OOM. Anyway additionally an explicit call to get the fill ratio will be
added in the future.
2018-04-11 16:22:52 +02:00
antirez a0149c995b getMaxmemoryState() fixed and improved. 2018-04-11 12:48:26 +02:00
artix f3c9a00c1c Cluster Manager: added clusterManagerCheckCluster to import command 2018-04-10 16:53:24 +02:00
artix 7866903555 Cluster Manager: import command 2018-04-10 16:25:25 +02:00
dejun.xdj e7c0edc734 Return more accurate message when specified ID is invalid 2018-04-10 17:11:10 +08:00
dejun.xdj 2e0d90fc96 Bugfix: xadd command ID parse
strictly check the string to be converted,
strtoull() in libc may not set errno to EINVAL when the string contains
invalid digits.
2018-04-10 17:05:48 +08:00
dejun.xdj 32c2185ff5 Bugfix: xadd comand <field,value> arity check
missing parenthesis causes wrong arithmetic priority.
2018-04-10 16:36:05 +08:00
antirez 0d46b971f6 Modules: context flags now include OOM flag.
Plus freeMemoryIfNeeded() refactoring to improve legibility.
Please review this commit for sanity.
2018-04-09 17:44:30 +02:00
antirez 01a2f09e18 freeMemoryIfNeeded() lacked a top comment. 2018-04-09 17:20:36 +02:00
antirez 095de965c1 Modules: remove trailing empty spaces. 2018-04-09 17:16:55 +02:00
antirez 86c9680425 Modules API: experimental APIs version.
This way it is possible to use conditional compilation to be compatible
with a larger amount of Redis versions, however note that this breaks
binary compatibiltiy, so the module must be compiled with the
corresponding redismodule.h file depending on the version of Redis
targeted.
2018-04-09 12:17:19 +02:00
antirez 73b4f7873a Modules API: blocked client free callback modified to get a context.
Note that this was an experimental API that can only be enabled with
REIDSMODULE_EXPERIMENTAL_API, so it is subject to change until its
promoted to stable API. Sorry for the breakage, it is trivial to
resolve btw. This change will not be back ported to Redis 4.0.
2018-04-09 11:54:44 +02:00
artix 00771be4e0 Cluster Manager: fix command. 2018-04-06 18:02:40 +02:00
charpty a3eeb76fe3 Remove unnecessary return statements
Signed-off-by: charpty <charpty@gmail.com>
2018-04-06 18:46:24 +08:00
antirez 7b028a237c Modules API: RM_GetRandomBytes() / GetRandomHexChars(). 2018-04-05 13:24:22 +02:00
antirez 978dd8a7f5 Modules Cluster API: GetClusterSize() added. 2018-04-01 16:36:32 +02:00
antirez 9302109e72 Modules Cluster API: GetMyClusterID() added. 2018-04-01 16:20:57 +02:00
antirez 534147a368 Modules Timer API: add example of API. 2018-03-31 10:12:34 +02:00
antirez 736c0fc956 Modules Timer API: Wait at least 1 ms per iteration. Convert to ms. 2018-03-31 10:11:30 +02:00
antirez e40b8e352e Modules Timer API: fix wrong raxRemove() key argument. 2018-03-31 09:58:43 +02:00
antirez d120cca2f5 Modules Timer API: fix infinite loop and export API. 2018-03-31 00:44:46 +02:00
antirez 203257f3d7 Modules Timer API: timer handling implemented. 2018-03-30 22:50:21 +02:00
antirez 432e3e0b1c Modules Timer API: initial implementation. 2018-03-30 20:40:35 +02:00
antirez a8857e6cba Modules Cluster API: node API exported, example improved. 2018-03-30 17:00:45 +02:00
antirez ac2094f975 Modules Cluster API: nodes list and info API. 2018-03-30 16:16:47 +02:00
antirez 088dac5743 Modules Cluster API: node information struct and flags. 2018-03-30 13:16:55 +02:00
antirez d7faf88f39 Modules Cluster API: make node IDs pointers constant. 2018-03-30 13:16:07 +02:00
antirez 11e653b85b Modules Cluster API: add a simple example module. 2018-03-30 12:49:45 +02:00
antirez 9aa6ac8867 Modules Cluster API: fix new API calls exporting. 2018-03-30 12:49:16 +02:00
antirez d0bf651ff9 Modules Cluster API: sending / receiving API first implementation. 2018-03-30 11:06:08 +02:00
zhaozhao.zz 87d33fba1a debug: avoid free client unexpectedly when reload & loadaof 2018-03-29 23:20:58 +08:00
antirez 3cb3ea3902 Modules Cluster API: message bus implementation. 2018-03-29 15:13:31 +02:00
zhaozhao.zz e0ca7cb729 adjust position of _dictNextPower in dictExpand 2018-03-29 17:36:15 +08:00
antirez aed9670c57 Fix ae.c when a timer finalizerProc adds an event.
While this feature is not used by Redis, ae.c implements the ability for
a timer to call a finalizer callback when an timer event is deleted.
This feature was bugged since the start, and because it was never used
we never noticed a problem. However Anthony LaTorre was using the same
library in order to implement a different system: he found a bug that he
describes as follows, and which he fixed with the patch in this commit,
sent me by private email:

    --- Anthony email ---

've found one bug in the current implementation of the timed events.
It's possible to lose track of a timed event if an event is added in
the finalizerProc of another event.

For example, suppose you start off with three timed events 1, 2, and
3. Then the linked list looks like:

3 -> 2 -> 1

Then, you run processTimeEvents and events 2 and 3 finish, so now the
list looks like:

-1 -> -1 -> 2

Now, on the next iteration of processTimeEvents it starts by deleting
the first event, and suppose this finalizerProc creates a new event,
so that the list looks like this:

4 -> -1 -> 2

On the next iteration of the while loop, when it gets to the second
event, the variable prev is still set to NULL, so that the head of the
event loop after the next event will be set to 2, i.e. after deleting
the next event the event loop will look like:

2

and the event with id 4 will be lost.

I've attached an example program to illustrate the issue. If you run
it you will see that it prints:

```
foo id = 0
spam!
```

But if you uncomment line 29 and run it again it won't print "spam!".

    --- End of email ---

Test.c source code is as follows:

    #include "ae.h"
    #include <stdio.h>

    aeEventLoop *el;

    int foo(struct aeEventLoop *el, long long id, void *data)
    {
	printf("foo id = %lld\n", id);

	return AE_NOMORE;
    }

    int spam(struct aeEventLoop *el, long long id, void *data)
    {
	printf("spam!\n");

	return AE_NOMORE;
    }

    void bar(struct aeEventLoop *el, void *data)
    {
	aeCreateTimeEvent(el, 0, spam, NULL, NULL);
    }

    int main(int argc, char **argv)
    {
	el = aeCreateEventLoop(100);

	//aeCreateTimeEvent(el, 0, foo, NULL, NULL);
	aeCreateTimeEvent(el, 0, foo, NULL, bar);

	aeMain(el);

	return 0;
    }

Anthony fixed the problem by using a linked list for the list of timers, and
sent me back this patch after he tested the code in production for some time.
The code looks sane to me, so committing it to Redis.
2018-03-28 14:11:04 +02:00
antirez 5ae0f745a5 Add INIT INFO to the provided init script. 2018-03-26 11:29:16 +02:00
antirez 5e831e92dd AOF: run tests with preamble off when it makes sense. 2018-03-25 13:03:38 +02:00
antirez d7c60b490b AOF: enable RDB-preamble rewriting by default.
There are too many advantages in doing this, RDB is faster to persist,
more compact, much faster to load back. The main issues here are that
the code is less tested because this was not the old default (so we are
enabling it for the new 5.0 release), and that the AOF is no longer a
trivially parsable format from now on. However the non-preamble mode
will be supported in the future as well, if new data types will be
added.
2018-03-25 11:43:30 +02:00
antirez ca339aef1b Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-03-23 17:21:46 +01:00
antirez a4b94b8e72 CG: AOF rewriting implemented. 2018-03-23 17:21:31 +01:00
artix 8b39e31ec3 Cluster Manager: rebalance command 2018-03-23 16:46:43 +01:00
Salvatore Sanfilippo db03b15e33 Merge pull request #4703 from gechunlin/unstable
Update object.c
2018-03-22 16:32:45 +01:00
Salvatore Sanfilippo b260f1fffb Merge pull request #4679 from youjiali1995/fix-lru
Some commands will udpate lfu-counter many times in one call.
2018-03-22 16:26:01 +01:00
Salvatore Sanfilippo 57f5f598cb Merge pull request #4722 from charsyam/feature/refactoring-call-aeDeleteFileEvent-twice-in-freeClusterLink
Refactoring to call aeDeleteFileEvent twice as once
2018-03-22 16:23:40 +01:00
Salvatore Sanfilippo 89be375da0 Merge pull request #4777 from soloestoy/avoid-double-close
anet: avoid double close
2018-03-22 16:22:32 +01:00
Salvatore Sanfilippo 1132ee99e3 Merge pull request #4781 from guybe7/block_list_notify
Make blocking list commands send keyspace notifications
2018-03-22 16:21:19 +01:00
antirez 81effdd09b Add top comments in two addReply*() functions. 2018-03-22 11:45:04 +01:00
antirez e785589e7b Massivily simplify addReply*() functions in networking.c 2018-03-22 11:42:50 +01:00
Guy Benoish fd0d2a5489 Make blocking list commands send keyspace notifications 2018-03-22 17:22:26 +07:00
Salvatore Sanfilippo 21a0ccd612 Merge pull request #4757 from charsyam/feature/fix-invalid-listpack-type-in-makefile
fix listpack.c to listpack.o in Makefile
2018-03-22 09:20:17 +01:00
Salvatore Sanfilippo ab3798a11d Merge pull request #3826 from charsyam/feature/fix_dlopen_leak
fix missed dlclose leak in module.c
2018-03-22 09:19:26 +01:00
Salvatore Sanfilippo 6e09787828 Merge pull request #4691 from oranagra/active_defrag_v2
Active defrag v2
2018-03-22 09:16:32 +01:00
Salvatore Sanfilippo 66368f2a91 Merge pull request #4779 from charsyam/feature/fix-streamPropagateXCLAIM
fix memory leak for streamPropagateXCLAIM
2018-03-22 09:02:37 +01:00
charsyam a152681256 fix memory leak for streamPropagateXCLAIM 2018-03-21 17:41:59 +09:00
zhaozhao.zz ee642f57f3 fix missed call on freeaddrinfo 2018-03-21 15:34:13 +08:00
zhaozhao.zz 9726f64d2b anet: avoid double close 2018-03-21 15:13:51 +08:00
antirez abe3502ea7 CG: propagate XACK by incrementing server.dirty.
This should be more than enough, even if in case of partial IDs that are
not found, we send all the IDs to the slave/AOF, but this is definitely
a corner case without bad effects if not some wasted space.
2018-03-20 17:53:45 +01:00
antirez 9c5dcfdd76 Streams: improve MEMORY USAGE computation, include CGs. 2018-03-20 17:50:40 +01:00
antirez d28f9eb1b9 CG: Make XINFO Great Again (and more Redis-ish).
With XINFO out of the blue I invented a new syntax for commands never
used in Redis in the past... Let's fix it and make it

        Great Again!!11one (TM)
2018-03-20 11:52:42 +01:00
antirez d7cf910a03 CG: Add JUSTID to XCLAIM propagation to reduce CPU usage. 2018-03-20 11:25:49 +01:00
antirez 2f3149e0a0 CG: Fix propagate() arg count in streamPropagateXCLAIM(). 2018-03-20 11:13:41 +01:00
antirez 947f75098c CG: XGROUP CREATE/DELCONUSMER replication. 2018-03-20 11:07:01 +01:00
antirez 378b5761d8 CG: Replication WIP 1: XREADGROUP and XCLAIM propagated as XCLAIM. 2018-03-19 18:02:19 +01:00
antirez eea056aefd Merge branch 'oertl-hyperloglog-improvement' into unstable 2018-03-16 16:58:16 +01:00
antirez ebb8e3f816 Aesthetic changes to PR #4749. 2018-03-16 16:57:53 +01:00
antirez 4bbc9a5091 Fix HyperLogLog test script for new redis-rb API. 2018-03-16 16:34:04 +01:00
antirez 59d92024a2 Merge branch 'hyperloglog-improvement' of https://github.com/oertl/redis into oertl-hyperloglog-improvement 2018-03-16 16:27:38 +01:00
antirez e70a229a70 RDB version 9. 2018-03-16 13:48:44 +01:00
antirez 97d13bee18 RDB: Implement future-proof module AUX data loading. 2018-03-16 13:47:10 +01:00
zhaozhao.zz 1ebcebe762 add rdb-save-incremental-fsync option in redis.conf 2018-03-16 00:48:59 +08:00
zhaozhao.zz 4f0e303ff1 rdb: incremental fsync when redis saves rdb 2018-03-16 00:44:50 +08:00
antirez 0f3086777c RDB: make RDB check aware of LFU/LRU opcodes. 2018-03-15 16:39:33 +01:00
antirez 417dd585d3 RDB: LRU/LFU branches missed continue. 2018-03-15 16:33:18 +01:00
antirez 8e728a1612 RDB: Ability to load LFU/LRU info. 2018-03-15 16:24:53 +01:00
antirez 923fbbd9ed RDB: Ability to save LFU/LRU info.
This is a big win for caching use cases, since on reloading Redis will
still have some idea about what is worth to evict and what not.
However this only solves part of the problem because the information is
only partially propagated to slaves (on write operations). Reads will
not affect slaves LFU and LRU counters, so after a failover the eviction
decisions are kinda random until keys start to collect some aging/freq info.

However since new slaves are initially populated via RDB file transfer,
this means that if we spin up a new slave from a master, and perform an
immediate manual failover (for instance in order to upgrade the master),
the slave will have eviction informations to use for some time.

The LFU/LRU info is persisted only if the maxmemory policy is set to one
of the relevant type, even if no actual "maxmemory"  memory limit is
set.
2018-03-15 13:15:55 +01:00
antirez 071ef69726 CG: XINFO STREAM. 2018-03-15 12:54:10 +01:00
antirez d05efa5b3b CG: XINFO GROUPS + output format changes.
XINFO is mainly an observability command that will be used more by
humans than computers, and even when used by computers it will be a very
low traffic command. For this reason the format was changed in order to
have field names. They'll consume some bandwidth and CPU cycles, but in
this context this is much better than having to understand what the
numbers in the output array are.
2018-03-15 12:54:10 +01:00
antirez c5507af63c CG: XINFO CONSUMERS implemented. 2018-03-15 12:54:10 +01:00
antirez 7afd470477 Streams: fix XREAD missing check for NULL object. 2018-03-15 12:54:10 +01:00
antirez 5c1f1f43f8 CG: throw an error if XREADGROUP is used without GROUP. 2018-03-15 12:54:10 +01:00
antirez c0259d6f04 CG: XGROUP DELCONSUMER implemented. 2018-03-15 12:54:10 +01:00
antirez edd91aeb36 CG: remove stale comment. XREADGROUP is now implemented. 2018-03-15 12:54:10 +01:00
antirez a372b4c3ec CG: implement RETRYCOUNT in XCLAIM. 2018-03-15 12:54:10 +01:00
antirez 6aabe0fde7 CG: fix CG RDB loading not found conditional. 2018-03-15 12:54:10 +01:00
antirez 3d4a904fe9 CG: NACK created by XCLAIM FORCE must be set in global PEL. 2018-03-15 12:54:10 +01:00
antirez 96fad23879 CG: implement XCLAIM FORCE option. 2018-03-15 12:54:10 +01:00
antirez 038488c0c4 CG: XCLAIM now updates the idle time of the message. 2018-03-15 12:54:10 +01:00
antirez 2966aac16d Streams: trap more errors in stream loading + RDB check type name. 2018-03-15 12:54:10 +01:00
antirez 18f8c34d4a CG: XCLAIM, use minidle and fix array len. 2018-03-15 12:54:10 +01:00
antirez 8348d92e56 CG: remove unused argument from streamReplyWithRangeFromConsumerPEL(). 2018-03-15 12:54:10 +01:00
antirez 431e12a033 CG: fix RDB saving when there are no consumer groups. 2018-03-15 12:54:10 +01:00
antirez 591e3f031d Streams: fix error description for XADD when specified ID is small. 2018-03-15 12:54:10 +01:00
antirez 096350cf33 CG: XCLAIM initial draft. 2018-03-15 12:54:10 +01:00
antirez bab33460ab Make addReplyError...() family functions able to get error codes.
Now you can use:

    addReplyError("-MYERRORCODE some message");

If the error code is omitted, the behavior is like in the past,
the generic -ERR will be used.
2018-03-15 12:54:10 +01:00
antirez 75ad54c9a2 CG: test XACK remaining features. 2018-03-15 12:54:10 +01:00
antirez c3396a6c8d CG: test XACK ability to remove items from the PELs. 2018-03-15 12:54:10 +01:00
antirez 08ea1440cf CG: test XPENDING ability to return pending items. 2018-03-15 12:54:10 +01:00
antirez b1aa4a5c8b CG: test XGROUPREAD abilities. 2018-03-15 12:54:10 +01:00
antirez e780693add CG: test group creation. 2018-03-15 12:54:10 +01:00
antirez dfff06af91 CG: More specific duplicated group error. 2018-03-15 12:54:10 +01:00
antirez 81f45896d0 CG: RDB loading, fix inverted conditional. 2018-03-15 12:54:10 +01:00
antirez 84667bff70 CG: RDB loading first implementation. 2018-03-15 12:54:10 +01:00
antirez 76b264dd7e CG: RDB saving part 2, consumers. 2018-03-15 12:54:10 +01:00
antirez 3763d5c164 CG: RDB saving part 1, metadata and PEL. 2018-03-15 12:54:10 +01:00
antirez ec0faf1bd9 CG: XPENDING should not create consumers and obey to count. 2018-03-15 12:54:10 +01:00
antirez c0cf16f988 CG: XPENDING with start/stop/count variant implemented. 2018-03-15 12:54:10 +01:00
antirez af40a25c52 CG: XPENDING without start/stop variant implemented. 2018-03-15 12:54:10 +01:00
antirez 505520ab09 CG: Now XREADGROUP + blocking operations work. 2018-03-15 12:54:10 +01:00
antirez 797d4df6fd CG: XACK should return zero when nothing is processed. 2018-03-15 12:54:10 +01:00
antirez c6ec6fb82a CG: XACK implementation. 2018-03-15 12:54:10 +01:00
antirez 34d41d8aa5 CG: XREADGROUP can fetch data from the consumer PEL. 2018-03-15 12:54:10 +01:00
antirez 68a8dbb256 CG: first draft of streamReplyWithRangeFromConsumerPEL(). 2018-03-15 12:54:10 +01:00
antirez 9292fee76f CG: Fix order of calls in streamReplyWithRange().
We need to check if we are going to serve the request via the PEL before
inserting a deferred array len in the client output buffer.
2018-03-15 12:54:10 +01:00
antirez 4a5eaaeb76 CG: creation of NACK entries in PELs. 2018-03-15 12:54:10 +01:00
antirez 4de6308f6e CG: fix XREADGROUP ">" special ID parsing due to missing "continue". 2018-03-15 12:54:10 +01:00
antirez 5fa401d531 CG: streamCompareID() + group last_id updating. 2018-03-15 12:54:10 +01:00
antirez 6648d60246 CG: consumer lookup + initial streamReplyWithRange() work to supprot CG. 2018-03-15 12:54:10 +01:00
antirez f7fbc91be3 CG: add & populate group+consumer in the blocking state. 2018-03-15 12:54:10 +01:00
antirez 2cf40e99f2 CG: fix parsing in XREADGROUP and streamLookupCG() NULL check. 2018-03-15 12:54:10 +01:00
antirez 7d3280143c CG: add XREADGROUP in the command table. 2018-03-15 12:54:10 +01:00
antirez 59eb392351 CG: XGROUPREAD group option parsing and groups lookup. 2018-03-15 12:54:10 +01:00
antirez 4e65528940 CG: fix raxFind() retval check in streamCreateCG(). 2018-03-15 12:54:10 +01:00
antirez 4925f7773c CG: data structures design + XGROUP CREATE implementation. 2018-03-15 12:54:10 +01:00
charsyam 484ed5a521 fix listpack.c to listpack.o in Makefile 2018-03-15 20:32:08 +09:00
Otmar Ertl 3c56cb9977 fixed compilation error when using clang as reported by michael-grunder 2018-03-14 21:00:06 +01:00
antirez 8bfc141157 Cluster: add test for the nofailover flag. 2018-03-14 16:30:32 +01:00
antirez f467e1fce3 Cluster: ability to prevent slaves from failing over their masters.
This commit, in some parts derived from PR #3041 which is no longer
possible to merge (because the user deleted the original branch),
implements the ability of slaves to have a special configuration
preventing that they try to start a failover when the master is failing.

There are multiple reasons for wanting this, and the feautre was
requested in issue #3021 time ago.

The differences between this patch and the original PR are the
following:

1. The flag is saved/loaded on the nodes configuration.
2. The 'myself' node is now flag-aware, the flag is updated as needed
   when the configuration is changed via CONFIG SET.
3. The flag name uses NOFAILOVER instead of NO_FAILOVER to be consistent
   with existing NOADDR.
4. The redis.conf documentation was rewritten.

Thanks to @deep011 for the original patch.
2018-03-14 14:01:38 +01:00
Oran Agra d66739931b Adding real allocator fragmentation to INFO and MEMORY command + active defrag test
other fixes / improvements:
- LUA script memory isn't taken from zmalloc (taken from libc malloc)
  so it can cause high fragmentation ratio to be displayed (which is false)
- there was a problem with "fragmentation" info being calculated from
  RSS and used_memory sampled at different times (now sampling them together)

other details:
- adding a few more allocator info fields to INFO and MEMORY commands
- improve defrag test to measure defrag latency of big keys
- increasing the accuracy of the defrag test (by looking at real grag info)
  this way we can use an even lower threshold and still avoid false positives
- keep the old (total) "fragmentation" field unchanged, but add new ones for spcific things
- add these the MEMORY DOCTOR command
- deduct LUA memory from the rss in case of non jemalloc allocator (one for which we don't "allocator active/used")
- reduce sampling rate of the rss and allocator info
2018-03-12 15:08:52 +02:00
Oran Agra eb83a56f8e active defrag v2
- big keys are not defragged in one go from within the dict scan
  instead they are scanned in parts after the main dict hash bucket is done.
- add latency monitor sample for defrag
- change default active-defrag-cycle-min to induce lower latency
- make active defrag start a new scan right away if needed, so it's easier
  (for the test suite) to detect when it's done
- make active defrag quick the current cycle after each db / big key
- defrag  some non key long term global allocations
- some refactoring for smaller functions and more reusable code
- during dict rehashing, one scan iteration of the dict, can end up scanning
  one bucket in the smaller dict and many many buckets in the larger dict.
  so waiting for 16 scan iterations before checking the time, may be much too long.
2018-03-12 15:07:43 +02:00
Otmar Ertl 86ad4e06b8 use all 64 bits of the hash value instead of 63 2018-03-11 09:18:00 +01:00
Otmar Ertl 09f818bc20 made constant static 2018-03-10 20:44:20 +01:00
Otmar Ertl bacac52eec improved definition of HLL_Q 2018-03-10 20:22:42 +01:00
Otmar Ertl 47b0cfcbd2 improved HyperLogLog cardinality estimation
based on method described in https://arxiv.org/abs/1702.01284
that does not rely on any magic constants
2018-03-10 20:13:21 +01:00
Otmar Ertl 3eaf0e07d2 replaced tab by spaces 2018-03-10 20:09:41 +01:00
Guy Benoish 0888eb4a00 Don't call sdscmp() with shared.maxstring or shared.minstring 2018-03-06 20:14:35 +07:00
artix 928640dee0 clusterManagerAddSlots: changed the way ADDSLOTS command is built 2018-03-06 13:06:04 +02:00
artix 422e41606b ClusterManager: fixed --cluster-from 'all' parsing 2018-03-02 17:06:50 +01:00
pan.liangp fb23cd0627 move get clients max buffer calculate into info clients command 2018-03-02 17:16:00 +08:00
antirez c277aaf3b4 Stream: update the listpack pointer in streamTrimByLength(). 2018-03-01 17:26:02 +01:00
antirez 8e0fbea741 Remove warning from lpGet snprintf(). 2018-03-01 15:26:27 +01:00
antirez 1db665cda9 redis-cli: fix missed unit in array. Change define name. 2018-03-01 15:06:41 +01:00
charsyam ef132d1337 refactoring-call-aeDeleteFileEvent-twice-in-freeClusterLink 2018-03-01 22:30:39 +09:00
charsyam 063e4b44c0 fix dlopen leak 2018-03-01 21:22:42 +09:00
Salvatore Sanfilippo 38ecac9dd0 Merge pull request #4714 from charsyam/feature/fix-out-of-index-range
[BugFix] Fix out of array index range for findBigKeys in redis-cli
2018-03-01 03:39:15 -08:00
antirez 99f94354a6 Actually use ae_flags to add AE_BARRIER if needed.
Many thanks to @Plasma that spotted this problem reviewing the code.
2018-02-28 18:03:51 +01:00
Artix a00157c44f Cluster Manager: fixed some memory error 2018-02-28 15:21:08 +01:00
artix b15f3515f3 Fixed memory write error in clusterManagerGetConfigSignature 2018-02-28 11:49:10 +01:00
artix 66548863a4 Cluster Manager: reshard command, fixed slots
parsing bug and other minor bugs.
2018-02-28 10:44:14 +01:00
Salvatore Sanfilippo c3934db151 Merge pull request #4715 from charsyam/feature/refactoring-make-condition-clear-for-rdb
[BugFix] fix calculation length in rdbSaveAuxField
2018-02-27 10:15:27 -08:00
antirez 550181a96b expireIfNeeded() needed a top comment documenting the behavior. 2018-02-27 16:44:43 +01:00
antirez 4db08588cc expireIfNeeded() comment: claim -> pretend. 2018-02-27 16:37:37 +01:00
charsyam 7bf2ef9dba refactoring-make-condition-clear-for-rdb 2018-02-27 21:55:20 +09:00
charsyam aecbdde3c0 fix-out-of-index-range-for-redis-cli-findbigkey 2018-02-27 21:46:19 +09:00
antirez b745b98f3d ae.c: insetad of not firing, on AE_BARRIER invert the sequence.
AE_BARRIER was implemented like:

    - Fire the readable event.
    - Do not fire the writabel event if the readable fired.

However this may lead to the writable event to never be called if the
readable event is always fired. There is an alterantive, we can just
invert the sequence of the calls in case AE_BARRIER is set. This commit
does that.
2018-02-27 13:06:42 +01:00
antirez 5a57c953c9 AOF: fix a bug that may prevent proper fsyncing when fsync=always.
In case the write handler is already installed, it could happen that we
serve the reply of a query in the same event loop cycle we received it,
preventing beforeSleep() from guaranteeing that we do the AOF fsync
before sending the reply to the client.

The AE_BARRIER mechanism, introduced in a previous commit, prevents this
problem. This commit makes actual use of this new feature to fix the
bug.
2018-02-27 13:06:42 +01:00
antirez 61da5a9da8 Cluster: improve crash-recovery safety after failover auth vote.
Add AE_BARRIER to the writable event loop so that slaves requesting
votes can't be served before we re-enter the event loop in the next
iteration, so clusterBeforeSleep() will fsync to disk in time.
Also add the call to explicitly fsync, given that we modified the last
vote epoch variable.
2018-02-27 13:06:42 +01:00
antirez 6c7d27c711 ae.c: introduce the concept of read->write barrier.
AOF fsync=always, and certain Redis Cluster bus operations, require to
fsync data on disk before replying with an acknowledge.
In such case, in order to implement Group Commits, we want to be sure
that queries that are read in a given cycle of the event loop, are never
served to clients in the same event loop iteration. This way, by using
the event loop "before sleep" callback, we can fsync the information
just one time before returning into the event loop for the next cycle.
This is much more efficient compared to calling fsync() multiple times.

Unfortunately because of a bug, this was not always guaranteed: the
actual way the events are installed was the sole thing that could
control. Normally this problem is hard to trigger when AOF is enabled
with fsync=always, because we try to flush the output buffers to the
socekt directly in the beforeSleep() function of Redis. However if the
output buffers are full, we actually install a write event, and in such
a case, this bug could happen.

This change to ae.c modifies the event loop implementation to make this
concept explicit. Write events that are registered with:

    AE_WRITABLE|AE_BARRIER

Are guaranteed to never fire after the readable event was fired for the
same file descriptor. In this way we are sure that data is persisted to
disk before the client performing the operation receives an
acknowledged.

However note that this semantics does not provide all the guarantees
that one may believe are automatically provided. Take the example of the
blocking list operations in Redis.

With AOF and fsync=always we could have:

    Client A doing: BLPOP myqueue 0
    Client B doing: RPUSH myqueue a b c

In this scenario, Client A will get the "a" elements immediately after
the Client B RPUSH will be executed, even before the operation is persisted.
However when Client B will get the acknowledge, it can be sure that
"b,c" are already safe on disk inside the list.

What to note here is that it cannot be assumed that Client A receiving
the element is a guaranteed that the operation succeeded from the point
of view of Client B.

This is due to the fact that the barrier exists within the same socket,
and not between different sockets. However in the case above, the
element "a" was not going to be persisted regardless, so it is a pretty
synthetic argument.
2018-02-27 13:06:42 +01:00
Salvatore Sanfilippo 3f379e3d70 Merge pull request #3828 from oranagra/sdsnewlen_pr
add SDS_NOINIT option to sdsnewlen to avoid unnecessary memsets.
2018-02-27 04:04:32 -08:00
antirez ac49eb0c8a Fix ziplist prevlen encoding description. See #4705. 2018-02-23 12:19:35 +01:00
gechunlin c857ac5840 Update object.c 2018-02-22 20:57:54 -06:00
artix 749e093591 Cluster Manager:
- Almost all Cluster Manager related code moved to
  the same section.
- Many macroes converted to functions
- Added various comments
- Little code restyling
2018-02-22 18:35:40 +01:00
artix 4a89b1b7c5 - Fixed bug in clusterManagerGetAntiAffinityScore
- Code improvements
2018-02-22 18:35:40 +01:00
artix 2e64e25ee1 Cluster Manager: colorized output 2018-02-22 18:35:40 +01:00
artix b0e77e6afa Cluster Manager: improved cleanup/error handling in various functions 2018-02-22 18:35:40 +01:00
artix 46465a7942 Cluster Manager: 'call' command. 2018-02-22 18:35:40 +01:00
artix 273ba95485 Cluster Manager: CLUSTER_MANAGER_NODE_CONNECT macro 2018-02-22 18:35:40 +01:00
artix bb958098f4 ClusterManager: added replicas count to clusterManagerNode 2018-02-22 18:35:40 +01:00
artix 8251a1b736 Cluster Manager: cluster is considered consistent if only one node has been found 2018-02-22 18:35:40 +01:00
artix c6e8eae7ae Cluster Manager: reply error catch for MEET command 2018-02-22 18:35:40 +01:00
artix 6b13b265e4 Cluster Manager: slots coverage check. 2018-02-22 18:35:40 +01:00
artix 7769aa0f98 - Cluster Manager: fixed various memory leaks
- Cluster Manager: fixed flags assignment in
  clusterManagerNodeLoadInfo
2018-02-22 18:35:40 +01:00
artix 8d343fa60b Added check for open slots (clusterManagerCheckCluster) 2018-02-22 18:35:40 +01:00
artix e674d6f6c0 Cluster Manager: 'create', 'info' and 'check' commands 2018-02-22 18:35:40 +01:00
artix 67065d00a1 Cluster Manager mode 2018-02-22 18:35:39 +01:00
antirez ee84fc714a Track number of logically expired keys still in memory.
This commit adds two new fields in the INFO output, stats section:

expired_stale_perc:0.34
expired_time_cap_reached_count:58

The first field is an estimate of the number of keys that are yet in
memory but are already logically expired. They reason why those keys are
yet not reclaimed is because the active expire cycle can't spend more
time on the process of reclaiming the keys, and at the same time nobody
is accessing such keys. However as the active expire cycle runs, while
it will eventually have to return to the caller, because of time limit
or because there are less than 25% of keys logically expired in each
given database, it collects the stats in order to populate this INFO
field.

Note that expired_stale_perc is a running average, where the current
sample accounts for 5% and the history for 95%, so you'll see it
changing smoothly over time.

The other field, expired_time_cap_reached_count, counts the number
of times the expire cycle had to stop, even if still it was finding a
sizeable number of keys yet to expire, because of the time limit.
This allows people handling operations to understand if the Redis
server, during mass-expiration events, is able to collect keys fast
enough usually. It is normal for this field to increment during mass
expires, but normally it should very rarely increment. When instead it
constantly increments, it means that the current workloads is using
a very important percentage of CPU time to expire keys.

This feature was created thanks to the hints of Rashmi Ramesh and
Bart Robinson from Twitter. In private email exchanges, they noted how
it was important to improve the observability of this parameter in the
Redis server. Actually in big deployments, the amount of keys that are
yet to expire in each server, even if they are logically expired, may
account for a very big amount of wasted memory.
2018-02-19 11:12:49 +01:00
antirez f4395e232b Remove non semantical spaces from module.c. 2018-02-15 21:41:03 +01:00
Salvatore Sanfilippo 1d0a91aecb Merge pull request #4479 from dvirsky/notify
Keyspace notifications API for modules
2018-02-15 21:36:32 +01:00
antirez 906b095592 Fix typo in notifyKeyspaceEvent() comment. 2018-02-15 21:33:06 +01:00
Dvir Volk 0690168116 Add doc comment about notification flags 2018-02-14 21:54:00 +02:00
Dvir Volk d3abc6e3ae Add REDISMODULE_NOTIFY_STREAM flag to support stream notifications 2018-02-14 21:50:42 +02:00
Dvir Volk 4991298fb0 Fix indentation and comment style in testmodule 2018-02-14 21:43:06 +02:00
Dvir Volk fbd0514a1f Use one static client for all keyspace notification callbacks 2018-02-14 21:40:10 +02:00
Dvir Volk c54aaca680 Remove the NOTIFY_MODULE flag and simplify the module notification flow if there aren't subscribers 2018-02-14 21:40:10 +02:00
Dvir Volk fa3b63fe82 Document flags for notifications 2018-02-14 21:38:58 +02:00
Dvir Volk 8e5174caeb removed some trailing whitespaces 2018-02-14 21:38:58 +02:00
Dvir Volk fa158da45d removed hellonotify.c 2018-02-14 21:38:58 +02:00
Dvir Volk 641fce93e7 fixed test 2018-02-14 21:38:58 +02:00
Dvir Volk 053941b983 finished implementation of notifications. Tests unfinished 2018-02-14 21:38:58 +02:00
Salvatore Sanfilippo ca9192e2fa Merge pull request #4685 from charsyam/refactoring/set_max_latency
Removing duplicated code to set max latency
2018-02-13 16:20:32 +01:00
charsyam a7bb8bb27c getting rid of duplicated code 2018-02-14 00:12:13 +09:00
antirez bc0c7045f4 More verbose logging when slave sends errors to master.
See #3832.
2018-02-13 16:01:31 +01:00
Salvatore Sanfilippo 1296894d25 Merge pull request #3832 from oranagra/slave_reply_to_master_pr
when a slave responds with an error on commands that come from master, log it
2018-02-13 15:55:26 +01:00
Salvatore Sanfilippo 028efd8242 Merge pull request #3745 from guybe7/unstable
enlarged buffer given to ld2string
2018-02-13 15:50:21 +01:00
antirez d4f1cbbd3a Make it explicit with a comment why we kill the old AOF rewrite.
See #3858.
2018-02-13 15:43:34 +01:00
Guy Benoish cfa0d361b7 rewriteAppendOnlyFileBackground() failure fix
It is possible to do BGREWRITEAOF even if appendonly=no. This is by design.
stopAppendonly() didn't turn off aof_rewrite_scheduled (it can be turned on
again by BGREWRITEAOF even while appendonly is off anyway).
After configuring `appendonly yes` it will see that the state is AOF_OFF,
there's no RDB fork, so it will do rewriteAppendOnlyFileBackground() which
will fail since the aof_child_pid is set (was scheduled and started by cron).

Solution:
stopAppendonly() will turn off the schedule flag (regardless of who asked for it).
startAppendonly() will terminate any existing fork and start a new one (so it is the most recent).
2018-02-13 15:41:06 +01:00
Salvatore Sanfilippo 3ff53a8228 Merge pull request #4684 from oranagra/latency_monitor_max
fix to latency monitor reporting wrong max latency
2018-02-13 15:31:11 +01:00
Oran Agra a151e92947 fix to latency monitor reporting wrong max latency
in some cases LATENCY HISTORY reported latency that was
higher than the max latency reported by LATENCY LATEST / DOCTOR
2018-02-13 15:58:40 +02:00
赵磊 04a3125f02 Remove updateLFU() in dbOverwrite(). 2018-02-11 21:02:07 +08:00
antirez 0b58392719 Rax updated to latest antirez/rax commit. 2018-02-02 11:10:18 +01:00
zhaozhao.zz 59eb641bdf config: handle special configuration "" for auth 2018-01-26 22:49:39 +08:00
Salvatore Sanfilippo 6314165e4d Merge pull request #4269 from jianqingdu/unstable
fix not call va_end() when syncWrite() failed
2018-01-24 10:55:25 +01:00
Salvatore Sanfilippo 7c41737e39 Merge pull request #4628 from mnunberg/patch-1
redismodule.h: Check ModuleNameBusy before calling it
2018-01-24 10:48:04 +01:00
antirez a8d678d26d Fix integration test NOREPLICAS error time dependent false positive. 2018-01-24 10:10:48 +01:00
Mark Nunberg eb80bec044 redismodule.h: Check ModuleNameBusy before calling it
Older versions might not have this function.
2018-01-23 10:49:18 -05:00
antirez bca90f9ef7 Fix migrateCommand() access of not initialized byte. 2018-01-18 12:41:05 +01:00
Guy Benoish 99e7e46da1 Replication buffer fills up on high rate traffic.
When feeding the master with a high rate traffic the the slave's feed is much slower.
This causes the replication buffer to grow (indefinitely) which leads to slave disconnection.
The problem is that writeToClient() decides to stop writing after NET_MAX_WRITES_PER_EVENT
writes (In order to be fair to clients).
We should ignore this when the client is a slave.
It's better if clients wait longer, the alternative is that the slave has no chance to stay in
sync in this situation.
2018-01-18 12:10:48 +01:00
antirez 0faed417e5 Cluster: improve anti-affinity algo in redis-trib.rb.
See #3462 and related PRs.

We use a simple algorithm to calculate the level of affinity violation,
and then an optimizer that performs random swaps until things improve.
2018-01-18 11:44:19 +01:00
antirez dbc3625a80 Remove useless comment from serverCron().
The behavior is well specified by the code itself.
2018-01-17 11:23:41 +01:00
Salvatore Sanfilippo a28f3fc568 Merge pull request #4546 from hqin6/unstable
fixbug for #4545 dead loop aof rewrite
2018-01-17 11:21:55 +01:00
heqin 28405f1f75 fixbug for #4545 dead loop aof rewrite 2018-01-17 18:08:30 +08:00
Salvatore Sanfilippo d19edaf43f Merge pull request #4609 from Qinch/unstable
fix assert problem in ZIP_DECODE_PREVLENSIZE macro
2018-01-17 10:45:11 +01:00
antirez 299e560a2e Hopefully more clear comment to explain the change in #4607. 2018-01-16 15:52:13 +01:00
qinchao f2db112964 fix assert problem in ZIP_DECODE_PREVLENSIZE
, see issue: https://github.com/antirez/redis/issues/4587
2018-01-16 22:43:06 +08:00
Salvatore Sanfilippo 887e242cea Merge pull request #4607 from oranagra/psync2_backlog
PSYNC2 fix - promoted slave should hold on to it's backlog
2018-01-16 15:32:58 +01:00
Oran Agra 09680b7c6f PSYNC2 fix - promoted slave should hold on to it's backlog
after a slave is promoted (assuming it has no slaves
and it booted over an hour ago), it will lose it's replication
backlog at the next replication cron, rather than waiting for slaves
to connect to it.
so on a simple master/slave faiover, if the new slave doesn't connect
immediately, it may be too later and PSYNC2 will fail.
2018-01-16 10:10:42 +02:00
zhaozhao.zz 1ebec13091 aof: format code and comment 2018-01-15 13:01:03 +01:00
antirez 402053e9fe Put more details in the comment introduced by #4601. 2018-01-15 12:50:08 +01:00
Salvatore Sanfilippo bcb5bc6742 Merge pull request #4601 from soloestoy/fix-memoryleak-for-lazy-server-del
lazyfree: fix memory leak for lazyfree-lazy-server-del
2018-01-15 12:43:55 +01:00
zhaozhao.zz 3f7e0f708f lazyfree: fix memory leak for lazyfree-lazy-server-del 2018-01-15 00:45:37 +08:00
Salvatore Sanfilippo 9e2a305918 Merge pull request #4575 from soloestoy/bugfix-benchmark
redis-benchmark: bugfix - handle zero liveclients in right way
2018-01-12 17:43:01 +01:00
Salvatore Sanfilippo 86fce15f71 Merge pull request #4581 from dvirsky/module_unlink
Added RM_UnlinkKey - a low level analog to UNLINK command
2018-01-12 17:41:09 +01:00
antirez cb00284d6e Merge branch 'unstable' of github.com:/antirez/redis into unstable 2018-01-12 17:16:38 +01:00
Salvatore Sanfilippo ccb1d5c533 Merge pull request #4586 from gnuhpc/fix-crashlog-typo
Fix a typo(maybe instruction?) in crash log
2018-01-12 17:16:12 +01:00
antirez 677e9dd036 Fix getKeysUsingCommandTable() in the case of nagative arity.
This fixes a crash with Redis Cluster when OBJECT is mis-used, because
getKeysUsingCommandTable() will call serverPanic() detecting we are
accessing an invalid argument in the case "OBJECT foo" is called.

This bug was introduced when OBJECT HELP was introduced, because the key
argument is set fixed at index 2 in the command table, however now
OBJECT may be called with an insufficient number of arguments to extract
the key.

The "Right Thing" would be to have a specific function to extract keys
from the OBJECT command, however this is kinda of an overkill, so I
preferred to make getKeysUsingCommandTable() more robust and just return
no keys when it's not possible to honor the command table, because new
commands are often added and also there are a number with an HELP
subcommand violating the normal form, and crashing for this trivial
reason or having many command-specific key extraction functions is not
great.
2018-01-12 11:26:29 +01:00
antirez 6c2197080c Document new protocol options in #4568 into redis.conf. 2018-01-11 11:40:57 +01:00
antirez 9fc237736e proto-max-querybuf-len -> client-query-buffer-limit.
We already had client buffer limits exported as configuration options.
Stick with the naming scheme already used.

See #4568.
2018-01-11 11:36:26 +01:00
antirez a3843bfe99 New config options about protocol prefixed with "proto".
Related to #4568.
2018-01-11 11:27:41 +01:00
Salvatore Sanfilippo a719121aa1 Merge pull request #4568 from oranagra/restore_size_limit
fix RESTORE command size limits
2018-01-11 11:20:02 +01:00
huijing.whj 3b332f4c57 fix int overflow problem in freeMemoryIfNeeded 2018-01-10 23:47:02 +08:00
Salvatore Sanfilippo e8dfc3f138 Merge pull request #3356 from yusaku/fix-module-firstkey
Fix the firstkey, lastkey, and keystep of moduleCommand
2018-01-09 18:58:31 +01:00
antirez 7eb5bdadcd Test: MIGRATE AUTH test added.
Related to #2507.
2018-01-09 18:49:19 +01:00
antirez 0c65678e6c Rewrite MIGRATE AUTH option.
See PR #2507. This is a reimplementation of the fix that contained
different problems.
2018-01-09 18:48:26 +01:00
gnuhpc fc7fd14395 Fix a typo(maybe instruction?) in crash log 2018-01-09 16:13:02 +08:00
Dvir Volk f47914ea05 Added RM_UnlinkKey - a low level analog to UNLINK command 2018-01-07 16:41:43 +02:00
Salvatore Sanfilippo 475054f8bd Merge pull request #4579 from gnuhpc/fix-memoryusage-list
Fix memory usage list bug
2018-01-05 11:53:11 +01:00
gnuhpc ec24b8bc6e Fix memory usage list bug 2018-01-05 12:16:24 +08:00
zhaozhao.zz 84af8a13bb redis-benchmark: bugfix - handle zero liveclients in right way 2018-01-03 12:47:25 +08:00
Oran Agra f15d20d9c7 Add config options for max-bulk-len and max-querybuf-len mainly to support RESTORE of large keys 2017-12-29 12:43:48 +02:00
Oran Agra 7fbdeedd6a fix processing of large bulks (above 2GB)
- protocol parsing (processMultibulkBuffer) was limitted to 32big positions in the buffer
  readQueryFromClient potential overflow
- rioWriteBulkCount used int, although rioWriteBulkString gave it size_t
- several places in sds.c that used int for string length or index.
- bugfix in RM_SaveAuxField (return was 1 or -1 and not length)
- RM_SaveStringBuffer was limitted to 32bit length
2017-12-29 12:24:19 +02:00
antirez 3e4795c48b Hyperloglog: refresh hdr variable correctly.
This is a fix for the #3819 improvements. The o->ptr may change because
of hllSparseSet() calls, so 'hdr' must be correctly re-fetched.
2017-12-22 11:26:31 +01:00
antirez 9da6cc96ed Hyperloglog: Support for PFMERGE sparse encoding as target.
This is a fix for #3819.
2017-12-22 11:01:27 +01:00
antirez ec89f279f8 Hyperloglog: refactoring of sparse/dense add function.
The commit splits the add functions into a set() and add() set of
functions, so that it's possible to set registers in an independent way
just having the index and count.

Related to #3819, otherwise a fix is not possible.
2017-12-22 11:00:22 +01:00
heqin fda7b81f8d fixbug for #4545 dead loop aof rewrite 2017-12-18 17:59:03 +08:00
Itamar Haber 43bf55b363 Uppercases subcommands in OBJECT HELP 2017-12-15 21:21:12 +02:00
Itamar Haber 847313fa93 Uppercases subcommands in MEMORY HELP 2017-12-15 21:19:41 +02:00
Salvatore Sanfilippo 026ced56b9 Merge pull request #4540 from hqin6/unstable
fixbug for #4538 Error opening /setting AOF rewrite IPC pipes: No suc…
2017-12-15 12:41:02 +01:00
heqin 0d47a71887 fixbug for #4538 Error opening /setting AOF rewrite IPC pipes: No such file or directory 2017-12-15 11:06:58 +08:00
Salvatore Sanfilippo 16ad748b26 Merge pull request #4536 from tporadowski/antirez_unstable-cli-history
Always enable command history in redis-cli run on a TTY
2017-12-14 22:14:45 +01:00
Salvatore Sanfilippo 7ef83ea2e9 Merge pull request #4528 from nashe/patch-oob-read-4527
Prevent off-by-one read in stringmatchlen()
2017-12-14 12:21:32 +01:00
antirez 2914126d41 safe_write -> aofWrite. Function commented.
Related to #4498.
2017-12-14 12:19:36 +01:00
Salvatore Sanfilippo 1c51d71eda Merge pull request #4498 from soloestoy/aof-safe-write
Aof safe write -- fix the short write
2017-12-14 12:15:46 +01:00
Tomasz Poradowski 15e36316ed always enable command history in redis-cli
- when redis-cli is running in a TTY - always enable command history
buffering, regardless if history file path can be successfully
determined
2017-12-13 23:49:41 +01:00
antirez 2de3d4c238 Cluster: allow read-only EVAL/EVALSHA in slaves.
Fix #3665.
2017-12-13 13:36:01 +01:00
nashe e255e8a23e Prevent off-by-one read in stringmatchlen() (fixes #4527) 2017-12-12 01:25:03 +01:00
Itamar Haber 4ec4a516f5 Standardizes MEMORY HELP subcommand 2017-12-10 17:54:56 +02:00
zhaozhao.zz 2d3bb10b74 zset: change the span of zskiplistNode to unsigned long 2017-12-08 16:09:27 +08:00
zhaozhao.zz b6b9199396 zset: fix the int problem 2017-12-08 15:37:08 +08:00
antirez c6243a942d Change indentation and other minor details of PR #4489.
The main change introduced by this commit is pretending that help
arrays are more text than code, thus indenting them at level 0. This
improves readability, and is an old practice when defining arrays of
C strings describing text.

Additionally a few useless return statements are removed, and the HELP
subcommand capitalized when printed to the user.
2017-12-06 12:05:14 +01:00
Itamar Haber 2d0c00ff5f C style 2017-12-05 19:09:19 +02:00
Itamar Haber 66e5f13e7a Uses an offset in addReplyHelp 2017-12-05 18:17:14 +02:00
Itamar Haber ae758ab46b Merge remote-tracking branch 'upstream/unstable' into help_subcommands 2017-12-05 18:14:59 +02:00
antirez 7bec849a07 add linkClient(): adds the client and caches the list node.
We have this operation in two places: when caching the master and
when linking a new client after the client creation. By having an API
for this we avoid incurring in errors when modifying one of the two
places forgetting the other. The function is also a good place where to
document why we cache the linked list node.

Related to #4497 and #4210.
2017-12-05 16:02:03 +01:00
Salvatore Sanfilippo 873752cd76 Merge pull request #4497 from soloestoy/optimize-unlink-client
networking: optimize unlinkClient() in freeClient()
2017-12-05 15:51:15 +01:00
Salvatore Sanfilippo 3f1bfac9c6 Merge pull request #4508 from trevor211/fixNotes
fix some notes
2017-12-05 15:47:05 +01:00
antirez 03b7d8d850 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2017-12-05 15:38:21 +01:00
zhaozhao.zz af08cd716d dict: fix the int problem for defrag 2017-12-05 15:38:03 +01:00
zhaozhao.zz c6ed4c399a dict: fix the int problem 2017-12-05 15:37:59 +01:00
Salvatore Sanfilippo 77ff594df3 Merge pull request #4488 from itamarhaber/debug_arity
Standardizes arity handling of DEBUG
2017-12-05 15:29:42 +01:00
Salvatore Sanfilippo 6c6ba30cfa Merge pull request #4509 from soloestoy/set-int-problem
Set int problem
2017-12-05 15:28:08 +01:00
zhaozhao.zz 470090fc4a set: fix the int problem for qsort 2017-12-05 17:42:19 +08:00
zhaozhao.zz 9a6a24d944 set: fix the int problem for SPOP & SRANDMEMBER 2017-12-05 17:19:19 +08:00
WuYunlong 6920f879b0 fix some notes 2017-12-05 14:41:16 +08:00
Salvatore Sanfilippo 7e16557875 Merge pull request #4506 from soloestoy/quicklist-int-problem
Quicklist int problem
2017-12-04 17:25:34 +01:00
zhaozhao.zz 68977f5138 quicklist: change the len of quicklist to unsigned long 2017-12-04 19:49:40 +08:00
zhaozhao.zz 549cca35ad quicklist: fix the return value of quicklistCount 2017-12-04 19:49:23 +08:00
antirez a135f4f52c Streams: fix a few type mismatches in t_stream.c. 2017-12-04 11:59:15 +01:00
antirez 967bc9e1c7 Refactoring: improve luaCreateFunction() API.
The function in its initial form, and after the fixes for the PSYNC2
bugs, required code duplication in multiple spots. This commit modifies
it in order to always compute the script name independently, and to
return the SDS of the SHA of the body: this way it can be used in all
the places, including for SCRIPT LOAD, without duplicating the code to
create the Lua function name. Note that this requires to re-compute the
body SHA1 in the case of EVAL seeing a script for the first time, but
this should not change scripting performance in any way because new
scripts definition is a rare event happening the first time a script is
seen, and the SHA1 computation is anyway not a very slow process against
the typical Redis script and compared to the actua Lua byte compiling of
the body.

Note that the function used to assert() if a duplicated script was
loaded, however actually now two times over three, we want the function
to handle duplicated scripts just fine: this happens in SCRIPT LOAD and
in RDB AUX "lua" loading. Moreover the assert was not defending against
some obvious failure mode, so now the function always tests against
already defined functions at start.
2017-12-04 11:25:20 +01:00
antirez f08a02dac5 Remove useless variable check from luaCreateFunction().
The block is already inside if (allow_dup).
2017-12-04 10:55:54 +01:00
antirez 78f8583063 Fix issue #4505, Lua RDB AUX field loading of existing scripts.
Unfortunately, as outlined by @soloestoy in #4505, "lua" AUX RDB field
loading in case of duplicated script was still broken. This commit fixes
this problem and also a memory leak introduced by the past commit.

Note that now we have a regression test able to duplicate the issue, so
this commit was actually tested against the regression. The original PR
also had a valid fix, but I prefer to hide the details of scripting.c
outside scripting.c, and later "SCRIPT LOAD" should also be able to use
the function luaCreateFunction() instead of redoing the work.
2017-12-04 10:33:04 +01:00
antirez f489746d4b Regression test for #4505 (Lua AUX field loading). 2017-12-04 10:26:02 +01:00
antirez 0ae7410b24 DEBUG change-repl-id implemented.
With PSYNC2 to force a full SYNC in tests is hard. With this new DEBUG
subcommand we just need to call it and then CLIENT KILL TYPE master in
the slave.
2017-12-04 10:24:52 +01:00
Itamar Haber fbf582220e Adds help to CONFIG 2017-12-03 19:34:31 +02:00
Itamar Haber 401bb5b6a9 Adds help to CLUSTER command 2017-12-03 19:05:10 +02:00
Itamar Haber d983ce1e4a Improve slowlog help 2017-12-03 17:39:52 +02:00
Itamar Haber f6666c2283 Helps CLIENT 2017-12-03 16:49:29 +02:00
antirez cfef19ea19 Fix loading of RDB files lua AUX fields when the script is defined.
In the case of slaves loading the RDB from master, or in other similar
cases, the script is already defined, and the function registering the
script should not fail in the assert() call.
2017-12-01 16:01:10 +01:00
antirez 69701e2fac Streams: DEBUG DIGEST support. 2017-12-01 15:04:05 +01:00
antirez d4d8154daf Streams: add code to compute the stream memory usage.
It's a bit of black magic without actually tracking it inside rax.c,
however Redis usage of the radix tree for the stream data structure is
quite consistent, so a few magic constants apparently are producing
results that make sense.
2017-12-01 12:50:27 +01:00
antirez 62836254be Streams: add some initial test for XREVRANGE. 2017-12-01 10:24:25 +01:00
antirez 620a9086db Streams: fix lp-count field for non-same-fields entries. 2017-12-01 10:24:25 +01:00
antirez 0c06581bee Streams: XRANGE REV option -> XREVRANGE command. 2017-12-01 10:24:25 +01:00
antirez c46aa40cca Streams: fix reverse iterator discarding of items out of range. 2017-12-01 10:24:25 +01:00
antirez 681efe652e Streams: fix reverse iteration next node jumping. 2017-12-01 10:24:25 +01:00
antirez fbee3af6ad Streams: state machine for reverse iteration WIP 1. 2017-12-01 10:24:25 +01:00
antirez 2ac8459187 Streams: augment stream entries to allow backward scanning. 2017-12-01 10:24:25 +01:00
antirez 77d514201c Streams: Update listpack to fix 32bit strings encoding error.
Note that streams produced by XADD in previous broken versions having
elements with 4096 bytes or more will be permanently broken and must be
created again from scratch.

Fix #4428
Fix #4349
2017-12-01 10:24:24 +01:00
antirez d42f4b1414 Streams: fix COUNT parsing, issue #4433. 2017-12-01 10:24:24 +01:00
antirez f065f551a9 Streams: fix redis-cli to understand the stream type. 2017-12-01 10:24:24 +01:00
antirez 97d57e3052 Streams: fix TYPE for stream type. 2017-12-01 10:24:24 +01:00
antirez 6c3b947799 Streams: fix XREAD test broken after previous tests improvements.
10% of times the data is not just "item 0" but there is also the
"otherfield" part. Use [lrange] to avoid the issue.
This commit fixes #4416.
2017-12-01 10:24:24 +01:00
antirez 503e3053ee Streams: move ID ms/seq separator from '.' to '-'
After checking with the community via Twitter (here:
https://twitter.com/antirez/status/915130876861788161) the verdict was to
use ":". However I later realized, after users lamented the fact that
it's hard to copy IDs just with double click, that this was the reason
why I moved to "." in the first instance. Fortunately "-", that was the
other option with most votes, also gets selected with double click on
most terminal applications on Linux and MacOS.

So my reasoning was:

1) We can't retain "." because it's actually confusing to newcomers, it
looks like a floating number, people may be tricked into thinking they
can order IDs numerically as floats.

2) Moving to a double-click-to-select format is much better. People will
work with such IDs for long time when coding / debugging. Why making now
a choice that will impact this for the next years?

The only other viable option was "-", and that's what I did. Thanks.
2017-12-01 10:24:24 +01:00
antirez 8493fd310e Streams: fix XADD + MAXLEN propagation due to var shadowing.
Clang should be more prone to return warnings by default when there is
same-var-name shadowing. GCC does this and can avoid bugs like that.
2017-12-01 10:24:24 +01:00
antirez 3f8034d75a Streams: fix memory leak in streamTrimByLength(). 2017-12-01 10:24:24 +01:00
antirez aa4a55ac97 Streams: add XADD + MAXLEN test. 2017-12-01 10:24:24 +01:00
antirez 6bb42f8d7f Streams: fix streamTrimByLength() standalone items skipping. 2017-12-01 10:24:24 +01:00
antirez 3eea06a2b3 Streams: XADD MAXLEN implementation.
The core of this change is the implementation of stream trimming, and
the resulting MAXLEN option of XADD as a trivial result of having
trimming functionalities. MAXLEN already works but in order to be more
efficient listpack GC should be implemented, currently marked as a TODO
item inside the comments.
2017-12-01 10:24:24 +01:00
antirez 51797185e0 Streams: reduce listpack max size to 2k to speedup range queries.
Listpack max size is a tradeoff between space and time. A 2k max entry
puts the memory usage approximately at a similar order of magnitude (5
million entries went from 96 to 120 MB), but the range queries speed
doubled (because there are half entries to scan in the average case).

Lower values could be considered, or maybe this parameter should be
made tunable.
2017-12-01 10:24:24 +01:00
antirez 79f540894a Streams: delta encode IDs based on key. Add count + deleted fields.
We used to have the master ID stored at the start of the listpack,
however using the key directly makes more sense in order to create a
space efficient representation: anyway the key at the radix tree is very
unlikely to change because of how the stream is implemented. Moreover on
nodes merging, to rewrite the merged listpacks is anyway the most
sensible operation, and we can use the iterator and the append-to-stream
function in order to avoid re-implementing the code needed for merging.

This commit also adds two items at the start of the listpack: the
number of valid items inside the listpack, and the number of items
marked as deleted. This means that there is no need to scan a listpack
in order to understand if it's a good candidate for garbage collection,
if the ration between valid/deleted items triggers the GC.
2017-12-01 10:24:24 +01:00
antirez 8538eacf16 Streams: specify better how the master enty works. 2017-12-01 10:24:24 +01:00
antirez e05a901cdc Streams: modify tests to stress compression. 2017-12-01 10:24:24 +01:00
antirez 731ad0ef1d Streams: items compression implemented.
The approach used is to set a fixed header at the start of every
listpack blob (that contains many entries). The header contains a
"master" ID and fields, that are initially just obtained from the first
entry inserted in the listpack, so that the first enty is always well
compressed. Later every new entry is checked against these fields, and
if it matches, the SAMEFIELD flag is set in the entry so that we know to
just use the master entry flags. The IDs are always delta-encoded
against the first entry. This approach avoids cascading effects in which
entries are encoded depending on the previous entries, in order to avoid
complexity and rewritings of the data when data is removed in the middle
(which is a planned feature).
2017-12-01 10:24:24 +01:00
antirez efd3268550 Streams: fixed memory leaks when blocking again for same stream.
blockForKeys() was not freeing the allocation holding the ID when the
key was already found busy. Fortunately the unit test checked explicitly
for blocking multiple times for the same key (copying a regression in
the blocking lists tests), so the bug was detected by the Redis test leak
checker.
2017-12-01 10:24:24 +01:00
antirez a2d7e004d4 Streams: tests for blocking and non-blocking XREAD. 2017-12-01 10:24:24 +01:00
antirez 6df222cbcd Streams: XRANGE fuzz testing. 2017-12-01 10:24:24 +01:00
antirez aa9cff0400 Streams: more advanced XADD and XRANGE tests. 2017-12-01 10:24:24 +01:00
antirez d9a50e2d94 Streams: basic XADD tests. 2017-12-01 10:24:24 +01:00
antirez 7118da8954 Streams: AOF rewriting + minor iterator improvements. 2017-12-01 10:24:24 +01:00
antirez 3d15e3f722 Streams: export iteration API. 2017-12-01 10:24:24 +01:00
antirez 1b0b5fa224 Streams: implement streamReplyWithRange() in terms of the iterator. 2017-12-01 10:24:24 +01:00
antirez e630b17282 Streams: stream iteration refactoring, WIP 2. 2017-12-01 10:24:24 +01:00
antirez 11045c4399 Streams: stream iteration refactoring, WIP 1. 2017-12-01 10:24:24 +01:00
antirez ba13aba3a0 Streams: fix bug in XREAD last received ID processing. 2017-12-01 10:24:24 +01:00
antirez 110e66b0c2 Streams: fix memory leak in freeStream(). 2017-12-01 10:24:24 +01:00
antirez 91e09d0959 Streams: rewrite XADD ID argument for AOF/slaves. 2017-12-01 10:24:24 +01:00
antirez eb8d0671f8 Streams: fix XADD API and keyspace notifications.
XADD was suboptimal in the first incarnation of the command, not being
able to accept an ID (very useufl for replication), nor options for
having capped streams.

The keyspace notification for streams was not implemented.
2017-12-01 10:24:24 +01:00
antirez 6410396372 Streams: When XREAD blocks without COUNT, set a default one.
A client may lose a lot of time between invocations of blocking XREAD,
for example because it is processing the messages or for any other
cause. When it returns back, it may provide a low enough message ID that
the server will block to send an unreasonable number of messages in a
single call. For this reason we set a COUNT when the client is blocked
with XREAD calls, even if no COUNT is given. This is arbitrarily set to
1000 because it's enough to avoid slowing down the reception of many
messages, but low enough to avoid to block.
2017-12-01 10:24:24 +01:00
antirez 789f74bba2 Streams: fix handleClientsBlockedOnKeys() access to invalid ID. 2017-12-01 10:24:24 +01:00
antirez 06a30111a8 Streams: fix XREAD ready-key signaling.
With lists we need to signal only on key creation, but streams can
provide data to clients listening at every new item added.
To make this slightly more efficient we now track different classes of
blocked clients to avoid signaling keys when there is nobody listening.
A typical case is when the stream is used as a time series DB and
accessed only by range with XRANGE.
2017-12-01 10:24:24 +01:00
antirez f3b3ca41f7 Streams: fix XREAD timeout handling, zero is valid. 2017-12-01 10:24:24 +01:00
antirez ea5ea8da3d Streams: XREAD related code to serve blocked clients. 2017-12-01 10:24:24 +01:00
antirez 6d54ec5e24 Streams: XREAD ability to block fixed. 2017-12-01 10:24:24 +01:00
antirez b6f1106630 Streams: synchronous xread fixes and improvements. 2017-12-01 10:24:24 +01:00
antirez bfeeeb9f34 Streams: XREAD get-key method fixed. 2017-12-01 10:24:24 +01:00
antirez 084a871019 Streams: XREAD get-keys method. 2017-12-01 10:24:24 +01:00
antirez 6f79b52dfa Streams: XREAD, first draft. Handling of blocked clients still missing. 2017-12-01 10:24:24 +01:00
antirez 54e221af43 Streams: XREAD arguments parsing. 2017-12-01 10:24:24 +01:00
antirez 428bb7cca0 Streams: augment client.bpop with XREAD specific fields. 2017-12-01 10:24:24 +01:00
antirez 6195349b50 Streams: more internal preparation for blocking XREAD. 2017-12-01 10:24:24 +01:00
antirez 896300be30 Streams: initial work to use blocking lists logic for streams XREAD. 2017-12-01 10:24:24 +01:00
antirez a168cbef13 Streams: implement stream object release. 2017-12-01 10:24:24 +01:00
antirez 760ad8f65c Streams: XLEN command. 2017-12-01 10:24:24 +01:00
antirez 64a7ad038d Streams: Save stream->length in RDB. 2017-12-01 10:24:24 +01:00
antirez fac8b716ab Streams: change listpack allocator to zmalloc. 2017-12-01 10:24:24 +01:00
antirez 90c980b105 Streams: RDB loading. RDB saving modified.
After a few attempts it looked quite saner to just add the last item ID
at the end of the serialized listpacks, instead of scanning the last
listpack loaded from head to tail just to fetch it. It's a disk space VS
CPU-and-simplicity tradeoff basically.
2017-12-01 10:24:24 +01:00
antirez f028e4e68e Streams: RDB saving. 2017-12-01 10:24:24 +01:00
antirez e4e216f8cc Streams: assign value of 6 to OBJ_STREAM + some refactoring. 2017-12-01 10:24:24 +01:00
antirez 71ad8ff26e Streams: 12 commits squashed into the initial Streams implementation. 2017-12-01 10:24:24 +01:00
antirez 95eaf3aada PSYNC2: Fix off by one buffer size in luaCreateFunction(). 2017-11-30 18:38:29 +01:00
antirez 6051ebea33 PSYNC2: just store script bodies into RDB.
Related to #4483. As suggested by @soloestoy, we can retrieve the SHA1
from the body. Given that in the new implementation using AUX fields we
ended copying around a lot to create new objects and strings, extremize
such concept and trade CPU for space inside the RDB file.
2017-11-30 18:38:26 +01:00
antirez d14beab331 PSYNC2: luaCreateFunction() should handle NULL client parameter.
See #4483. This is needed because luaCreateFunction() is now called
from RDB loading code outside a client context.
2017-11-30 18:37:52 +01:00
antirez 445d37f24d PSYNC2: Save Lua scripts state into RDB file.
This is currently needed in order to fix #4483, but this can be
useful in other contexts, so maybe later we may want to remove the
conditionals and always save/load scripts.

Note that we are using the "lua" AUX field here, in order to guarantee
backward compatibility of the RDB file. The unknown AUX fields must be
discarded by past versions of Redis.
2017-11-30 18:37:52 +01:00
antirez fbc1d53416 Regression test: Slave restart with EVALSHA in backlog issue #4483. 2017-11-30 18:37:10 +01:00
antirez 7f04efa1a8 Prevent corruption of server.executable after DEBUG RESTART.
Doing the following ended with a broken server.executable:

1. Start Redis with src/redis-server
2. Send CONFIG SET DIR /tmp/
3. Send DEBUG RESTART

At this point we called execve with an argv[0] that is no longer related
to the new path. So after the restart the absolute path of the
executable is recomputed in the wrong way. With this fix we pass the
absolute path already computed as argv[0].
2017-11-30 18:30:06 +01:00
antirez f8d42c5ec6 Be more verbose when DEBUG RESTART fails. 2017-11-30 18:08:21 +01:00
zhaozhao.zz 44e3dc083a networking: optimize unlinkClient() in freeClient() 2017-11-30 18:11:05 +08:00
zhaozhao.zz e40beab163 aof: cast sdslen to ssize_t 2017-11-30 10:27:12 +08:00
zhaozhao.zz ba78815203 aof: fix the short write 2017-11-30 10:22:12 +08:00
Itamar Haber 22698eef45 Check arity in SLOWLOG before accessing arg 2017-11-30 00:30:30 +02:00
antirez 7435fd6be9 Merge branch 'lfu-fixes' into unstable 2017-11-29 17:16:13 +01:00
Itamar Haber 0bc9d1d6f1 Merge branch 'unstable' into help_subcommands 2017-11-28 21:28:40 +02:00
Itamar Haber 78aabf66ff Standardizes the 'help' subcommand
This adds a new `addReplyHelp` helper that's used by commands
when returning a help text. The following commands have been
touched: DEBUG, OBJECT, COMMAND, PUBSUB, SCRIPT and SLOWLOG.

WIP

Fix entry command table entry for OBJECT for HELP option.

After #4472 the command may have just 2 arguments.

Improve OBJECT HELP descriptions.

See #4472.

WIP 2

WIP 3
2017-11-28 21:15:45 +02:00
Salvatore Sanfilippo e13b5ea614 Merge pull request #4200 from jeesyn/fix_typo
fix a typo
2017-11-28 18:44:11 +01:00
Salvatore Sanfilippo 5a836dc690 Merge pull request #4165 from zyegfryed/patch-1
Fix some typos
2017-11-28 18:43:45 +01:00
Salvatore Sanfilippo c847a62be6 Merge pull request #4166 from charpty/wip-redisclic-typo
redis-cli.c typo: helpe -> helper.
2017-11-28 18:41:51 +01:00
Salvatore Sanfilippo 35b359cdeb Merge pull request #4167 from charpty/wip-redisclic-typo2
redis-cli.c typo: Requets -> Requests.
2017-11-28 18:41:28 +01:00
Salvatore Sanfilippo 8f1d602a58 Merge pull request #4170 from TehWebby/patch-2
Fix typo
2017-11-28 18:40:43 +01:00
Salvatore Sanfilippo 4f5a9c99f8 Merge pull request #4172 from TehWebby/patch-3
Fix typo
2017-11-28 18:40:02 +01:00
Salvatore Sanfilippo 742d555754 Merge pull request #4176 from KrauseFx/patch-1
Update link to https and use inline link
2017-11-28 18:39:43 +01:00
antirez ff2b31e9a3 t_hash.c: clarify calling two times the same function. 2017-11-28 18:39:00 +01:00
antirez 788710fa44 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2017-11-28 18:27:35 +01:00
antirez 0cd79860ec adlist: fix listJoin() in the case the second list is empty.
See #4192, the original PR removed lines of code that are actually
needed, so thanks to @chunqiulfq for reporting the problem, but merging
solution from @jeesyn after checking, together with @artix75, that the
logic covers all the cases.
2017-11-28 18:25:14 +01:00
Salvatore Sanfilippo 7bb1fe976b Merge pull request #4215 from lamby/correct-faield-spelling
Correct spelling of "faield".
2017-11-28 18:08:32 +01:00
Salvatore Sanfilippo 355ba680e6 Merge pull request #4374 from rouzier/patch-1
Fix file descriptor leak and error handling
2017-11-28 17:33:23 +01:00
Salvatore Sanfilippo 1da0724a98 Merge pull request #4451 from devnexen/minor_build_fixes
Fix undefined behavior constant defined.
2017-11-28 17:23:48 +01:00
Itamar Haber 5919c4d1e9 Standardizes arity handling of DEBUG 2017-11-28 18:18:45 +02:00
antirez ac58cbc764 LFU: Fix LFUDecrAndReturn() to just decrement.
Splitting the popularity in half actually just needs decrementing the
counter because the counter is logarithmic.
2017-11-28 12:18:30 +01:00
zhaozhao.zz 0a76b9bcd8 LFU: add hotkeys option to redis-cli 2017-11-27 18:39:29 +01:00
zhaozhao.zz 660f01011c LFU: do some changes about LFU to find hotkeys
Firstly, use access time to replace the decreas time of LFU.
For function LFUDecrAndReturn,
it should only try to get decremented counter,
not update LFU fields, we will update it in an explicit way.
And we will times halve the counter according to the times of
elapsed time than server.lfu_decay_time.
Everytime a key is accessed, we should update the LFU
including update access time, and increment the counter after
call function LFUDecrAndReturn.
If a key is overwritten, the LFU should be also updated.
Then we can use `OBJECT freq` command to get a key's frequence,
and LFUDecrAndReturn should be called in `OBJECT freq` command
in case of the key has not been accessed for a long time,
because we update the access time only when the key is read or
overwritten.
2017-11-27 18:39:22 +01:00
zhaozhao.zz aef50770ba LFU: change lfu* parameters to int 2017-11-27 18:38:55 +01:00
zhaozhao.zz 44c2e4cdbf LFU: fix the missing of config get and rewrite 2017-11-27 18:38:33 +01:00
antirez d0a716f709 Improve OBJECT HELP descriptions.
See #4472.
2017-11-27 18:09:08 +01:00
antirez af514a1155 Fix entry command table entry for OBJECT for HELP option.
After #4472 the command may have just 2 arguments.
2017-11-27 13:16:07 +01:00
Salvatore Sanfilippo 922fc6bf1d Merge pull request #4472 from itamarhaber/object_patch
A minor fix and `help` subcommand for `OBJECT`
2017-11-27 12:41:02 +01:00
Itamar Haber 241526af3d Adds -u <uri> option to redis-cli. 2017-11-27 11:34:11 +01:00
Itamar Haber 4a9fb02aca Adds OBJECT help 2017-11-24 19:59:05 +02:00
Itamar Haber 21479f9ebc Prevents OBJECT freq with noeviction
When maxmemory is set to noeviction, idletime is implicitly kept. This renders access frequency nonsensical.
2017-11-24 19:58:37 +02:00
antirez f56b7aaef2 Test: regression test for latency expire events logging bug.
Regression for #4452.
2017-11-24 18:33:31 +01:00
Salvatore Sanfilippo 555a6ce980 Merge pull request #4452 from soloestoy/expire-latency
expire & latency: fix the missing latency records generated by expire
2017-11-24 18:21:35 +01:00
antirez 63f2530a04 Modules: fix memory leak in RM_IsModuleNameBusy(). 2017-11-24 13:29:54 +01:00
antirez 97ca53e8ab PSYNC2: reorganize comments related to recent fixes.
Related to PR #4412 and issue #4407.
2017-11-24 11:08:29 +01:00
Salvatore Sanfilippo 0cd51d21b3 Merge pull request #4412 from soloestoy/bugfix-psync2
PSYNC2: safe free backlog when reach the time limit and others
2017-11-24 10:56:18 +01:00
Salvatore Sanfilippo 4394eedf5e Merge pull request #4344 from soloestoy/fix-module-name-conflict
Fix module name conflict
2017-11-24 09:37:06 +01:00
Salvatore Sanfilippo b6b3811914 Merge pull request #4470 from oranagra/fix_string_to_double
fix string to double conversion, stopped parsing on \0 …
2017-11-24 08:59:23 +01:00
Oran Agra 07e0f0f72f fix string to double conversion, stopped parsing on \0 even if the string has more data.
getLongLongFromObject calls string2ll which has this line:
/* Return if not all bytes were used. */
so if you pass an sds with 3 characters "1\01" it will fail.

but getLongDoubleFromObject calls strtold, and considers it ok if eptr[0]==`\0`
i.e. if the end of the string found by strtold ends with null terminator

127.0.0.1:6379> set a 1
OK
127.0.0.1:6379> setrange a 2 2
(integer) 3
127.0.0.1:6379> get a
"1\x002"
127.0.0.1:6379> incrbyfloat a 2
"3"
127.0.0.1:6379> get a
"3"
2017-11-23 17:15:27 +02:00
antirez cff7b04fdc Modules: fix for scripting replication of modules commands.
See issue #4466 / #4467.
2017-11-23 15:14:17 +01:00
Salvatore Sanfilippo bf2b03202f Merge pull request #4467 from yossigo/fix-nested-multi
Nested MULTI/EXEC may replicate in different cases.
2017-11-23 13:38:43 +01:00
Yossi Gottlieb 3246b95129 Nested MULTI/EXEC may replicate in different cases.
For example:
1. A module command called within a MULTI section.
2. A Lua script with replicate_commands() called within a MULTI section.
3. A module command called from a Lua script in the above context.
2017-11-22 22:02:51 +02:00
zhaozhao.zz 308fa486ed PSYNC2: persist cached_master's dbid inside the RDB 2017-11-22 12:11:26 +08:00
zhaozhao.zz 7f0a2494ca PSYNC2: make repl_stream_db never be -1
it means that after this change all the replication
info in RDB is valid, and it can distinguish us from
the older version.
2017-11-22 12:05:34 +08:00
zhaozhao.zz 91922131ce expire & latency: fix the missing latency records generated by expire 2017-11-21 23:35:30 +08:00
zhaozhao.zz 32a1a4da1e rehash: handle one db until finished 2017-11-21 09:49:42 +01:00
David Carlier 7d9209e38b Fix undefined behavior constant defined. 2017-11-19 16:23:42 +00:00
Salvatore Sanfilippo 23fbdd5fd6 Merge pull request #2741 from kmiku7/unstable
fix boundary case for _dictNextPower
2017-11-08 17:06:09 +01:00
Salvatore Sanfilippo 4c308f7062 Merge pull request #4431 from itamarhaber/patch-10
Fixes an off-by-one in argument handling of `MEMORY USAGE`
2017-11-08 15:42:45 +01:00
Itamar Haber 0a8dfc8935 Fixes an off-by-one in argument handling of MEMORY USAGE
Fixes #4430
2017-11-08 16:08:29 +02:00
antirez 50ea884353 Fix saving of zero-length lists.
Normally in modern Redis you can't create zero-len lists, however it's
possible to load them from old RDB files generated, for instance, using
Redis 2.8 (see issue #4409). The "Right Thing" would be not loading such
lists at all, but this requires to hook in rdb.c random places in a not
great way, for a problem that is at this point, at best, minor.

Here in this commit instead I just fix the fact that zero length lists,
materialized as quicklists with the first node set to NULL, were
iterated in the wrong way while they are saved, leading to a crash.

The other parts of the list implementation are apparently able to deal
with empty lists correctly, even if they are no longer a thing.
2017-11-06 12:37:03 +01:00
antirez f8d379424e SDS: improve sdsRemoveFreeSpace() to avoid useless data copy.
Since SDS v2, we no longer have a single header, so the function to
rewrite the SDS in terms of the minimum space required, instead of just
using realloc() and let the underlying allocator decide what to do,
was doing an allocation + copy every time the minimum possible header
needed to represent the string was different than the current one.
This could be often a bit wasteful, because if we go, for instance, from
the 32 bit fields header to the 16 bit fields header, the overhead of
the header is normally very small. With this commit we call realloc
instead, unless the change in header size is very significant in relation
to the string length.
2017-11-03 10:19:27 +01:00
zhaozhao.zz db1d277847 PSYNC2: clarify the scenario when repl_stream_db can be -1 2017-11-02 10:45:33 +08:00
zhaozhao.zz 015a0f42b0 PSYNC2 & RDB: fix the missing rdbSaveInfo for BGSAVE 2017-11-01 17:52:43 +08:00
zhaozhao.zz 39acc4df0e PSYNC2: safe free backlog when reach the time limit
When we free the backlog, we should use a new
replication ID and clear the ID2. Since without
backlog we can not increment master_repl_offset
even do write commands, that may lead to inconsistency
when we try to connect a "slave-before" master
(if this master is our slave before, our replid
equals the master's replid2). As the master have our
history, so we can match the master's replid2 and
second_replid_offset, that make partial sync work,
but the data is inconsistent.
2017-11-01 17:32:27 +08:00
antirez 401cfa554f Fix buffer overflows occurring reading redis.conf.
There was not enough sanity checking in the code loading the slots of
Redis Cluster from the nodes.conf file, this resulted into the
attacker's ability to write data at random addresses in the process
memory, by manipulating the index of the array. The bug seems
exploitable using the following techique: the config file may be altered so
that one of the nodes gets, as node ID (which is the first field inside the
structure) some data that is actually executable: then by writing this
address in selected places, this node ID part can be executed after a
jump. So it is mostly just a matter of effort in order to exploit the
bug. In practice however the issue is not very critical because the
bug requires an unprivileged user to be able to modify the Redis cluster
nodes configuration, and at the same time this should result in some
gain. However Redis normally is unprivileged as well. Yet much better to
have this fixed indeed.

Fix #4278.
2017-10-31 09:41:22 +01:00
antirez a07c0db3af Regression test for issue #4391. 2017-10-30 13:45:46 +01:00
antirez fa36221bf6 More robust object -> double conversion.
Certain checks were useless, at the same time certain malformed inputs
were accepted without problems (emtpy strings parsed as zero).
Cases where strtod() returns ERANGE but we still want to parse the input
where ok in getDoubleFromObject() but not in the long variant.

As a side effect of these fixes, this commit fixes #4391.
2017-10-30 13:39:58 +01:00
rouzier 8eb5823f43 Fix file descriptor leak and error handling 2017-10-13 13:20:45 -04:00
antirez ba954302f6 Limit statement in RM_BlockClient() to 80 cols. 2017-09-28 23:15:34 +02:00
Salvatore Sanfilippo 32c0b27765 Merge pull request #4337 from dvirsky/module_ctx_flags
Module Context flags
2017-09-28 14:40:55 +02:00
zhaozhao.zz f1333eb992 Modules: handle the busy module name 2017-09-28 17:38:40 +08:00
zhaozhao.zz 7fb3864e95 Modules: handle the conflict of registering commands 2017-09-28 16:21:21 +08:00
Dvir Volk 6f37695620 Added safety net preventing redis from crashing if a module decide to block in MULTI 2017-09-27 15:17:53 +03:00
Dvir Volk c3777932f6 Renamed GetCtxFlags to GetContextFlags 2017-09-27 11:58:16 +03:00
Dvir Volk f5909f2c18 Added support for module context flags with RM_GetCtxFlags 2017-09-27 11:58:07 +03:00
antirez ab601f284f Clarify comment in change fixing #4323. 2017-09-21 12:35:04 +02:00
Salvatore Sanfilippo 52b92bc6ab Merge pull request #4324 from soloestoy/lazyfree-fix-memory-leak
Lazyfree: avoid memory leak when free slowlog entry
2017-09-21 12:32:02 +02:00
zhaozhao.zz 36252140aa Lazyfree: avoid memory leak when free slowlog entry 2017-09-21 14:19:21 +08:00
antirez 4e01e51338 PSYNC2: More refinements related to #4316. 2017-09-20 11:28:13 +02:00
zhaozhao.zz a45d0fc482 PSYNC2: make persisiting replication info more solid
This commit is a reinforcement of commit c1c99e9.

1. Replication information can be stored when the RDB file is
generated by a mater using server.slaveseldb when server.repl_backlog
is not NULL, or set repl_stream_db be -1. That's safe, because
NULL server.repl_backlog will trigger full synchronization,
then master will send SELECT command to replicaiton stream.
2. Only do rdbSave* when rsiptr is not NULL,
if we do rdbSave* without rdbSaveInfo, slave will miss repl-stream-db.
3. Save the replication informations also in the case of
SAVE command, FLUSHALL command and DEBUG reload.
2017-09-20 11:18:10 +02:00
antirez bc6b7c949d PSYNC2: Fix the way replication info is saved/loaded from RDB.
This commit attempts to fix a number of bugs reported in #4316.
They are related to the way replication info like replication ID,
offsets, and currently selected DB in the master client, are stored
and loaded by Redis. In order to avoid inconsistencies the changes in
this commit try to enforce that:

1. Replication information are only stored when the RDB file is
generated by a slave that has a valid 'master' client, so that we can
always extract the currently selected DB.
2. When replication informations are persisted in the RDB file, all the
info for a successful PSYNC or nothing is persisted.
3. The RDB replication informations are only loaded if the instance is
configured as a slave, otherwise a master can start with IDs that relate
to a different history of the data set, and stil retain such IDs in the
future while receiving unrelated writes.
2017-09-19 23:03:39 +02:00
antirez 9290786e73 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2017-09-19 10:35:49 +02:00
antirez d0c77132d8 PSYNC2: Create backlog on slave partial sync as well.
A slave may be started with an RDB file able to provide enough slave to
perform a successful partial SYNC with its master. However in such a
case, how outlined in issue #4268, the slave backlog will not be
started, since it was only initialized on full syncs attempts. This
creates different problems with successive PSYNC attempts that will
always result in full synchronizations.

Thanks to @fdingiit for discovering the issue.
2017-09-19 10:33:14 +02:00
Salvatore Sanfilippo a85314ecd3 Merge pull request #3785 from GitHubMota/unstable
redis-benchmark: default value size usage update.
2017-09-18 12:18:57 +02:00
Salvatore Sanfilippo ef7c5887b8 Merge pull request #3554 from jybaek/Delete_duplicate
Remove Duplicate Processing
2017-09-18 12:18:15 +02:00
Salvatore Sanfilippo 42e1112109 Merge pull request #4311 from oranagra/aof_shutdown_flush
Flush append only buffers before existing.
2017-09-18 12:03:41 +02:00
Oran Agra 000ae23df4 Flush append only buffers before existing.
when SHUTDOWN command is recived it is possible that some of the recent
command were not yet flushed from the AOF buffer, and the server
experiences data loss at shutdown.
2017-09-17 07:22:16 +03:00
jianqingdu 2f5239b038 fix not call va_end when syncWrite() failed
fix not call va_end when syncWrite() failed in sendSynchronousCommand()
2017-08-30 21:20:14 -05:00
Richard Li 572d8e0609 Update the comment 2017-08-18 11:27:45 +08:00
Chris Lamb 3666698eeb Correct spelling of "faield". 2017-08-12 22:21:03 -07:00
jeesyn.liu 3a207568cf fix a typo 2017-08-08 17:45:51 +08:00
Salvatore Sanfilippo e7b19b400a Merge pull request #4191 from jybaek/unstable
Add missing fclose()
2017-08-03 12:18:58 +02:00
jybaek 468c15935c Add missing fclose() 2017-08-03 17:28:04 +09:00
Salvatore Sanfilippo 6e3e80d10f Merge pull request #3935 from itamarhaber/module-cmdstats
Changes command stats iteration to being dict-based
2017-08-02 12:51:26 +02:00
Felix Krause 69adaf75de Update link to https and use inline link 2017-07-28 13:04:52 -04:00
antirez 94a549d660 Add MEMORY DOCTOR to MEMORY HELP. 2017-07-28 17:47:54 +02:00
Shaun Webb 17b80429dd Typo fix 2017-07-27 15:27:46 +09:00
Shaun Webb 4ef0fb2158 Fix typo 2017-07-27 09:37:37 +09:00
Bo Cai c9f7f77457 redis-cli.c typo: Requets -> Requests.
Signed-off-by: Bo Cai <charpty@gmail.com>
2017-07-26 21:33:29 +08:00
Bo Cai c58e1199d8 redis-cli.c typo: helpe -> helper.
Signed-off-by: Bo Cai <charpty@gmail.com>
2017-07-26 21:24:28 +08:00
Sébastien Fievet 38a9c8c6c0 Fix some typos 2017-07-26 14:11:05 +02:00
Mota b4ea09446f redis-benchmark: default value size usage update.
default size of SET/GET value in usage should be 3 bytes as in main code.
2017-07-25 23:43:46 +08:00
Salvatore Sanfilippo f82598303a Merge pull request #1934 from badboy/install-script-1922
Don't use extended Regexp Syntax
2017-07-24 15:20:31 +02:00
Salvatore Sanfilippo 3289f393b5 Merge pull request #2259 from badboy/fix-2258
Check that the whole first argument is a number
2017-07-24 15:19:53 +02:00
Salvatore Sanfilippo 5bb9eda3d4 Merge pull request #4124 from lamby/proceding-proceeding-typo
Correct proceding -> proceeding typo.
2017-07-24 15:19:21 +02:00
Salvatore Sanfilippo 37d8f254df Merge pull request #4125 from trevor211/fixAutoAofRewirteMinSize
fix rewrite config: auto-aof-rewrite-min-size
2017-07-24 15:18:56 +02:00
Salvatore Sanfilippo dc73925823 Merge pull request #1998 from grobe0ba/unstable
Fix missing '-' in redis-benchmark help output (Issue #1996)
2017-07-24 15:18:08 +02:00
Salvatore Sanfilippo 632c1a112f Merge pull request #4128 from leonchen83/unstable
fix mismatch argument and return wrong value of clusterDelNodeSlots
2017-07-24 14:18:28 +02:00
Salvatore Sanfilippo cf6e3bebed Merge pull request #4154 from liangsijian/unstable
Fix lua ldb command log
2017-07-24 14:11:10 +02:00
liangsijian 50e96b1488 Fix lua ldb command log 2017-07-24 19:24:06 +08:00
antirez 4bd29fcdab Modules: don't crash when Lua calls a module blocking command.
Lua scripting does not support calling blocking commands, however all
the native Redis commands are flagged as "s" (no scripting flag), so
this is not possible at all. With modules there is no such mechanism in
order to flag a command as non callable by the Lua scripting engine,
moreover we cannot trust the modules users from complying all the times:
it is likely that modules will be released to have blocking commands
without such commands being flagged correctly, even if we provide a way to
signal this fact.

This commit attempts to address the problem in a short term way, by
detecting that a module is trying to block in the context of the Lua
scripting engine client, and preventing to do this. The module will
actually believe to block as usually, but what happens is that the Lua
script receives an error immediately, and the background call is ignored
by the Redis engine (if not for the cleanup callbacks, once it
unblocks).

Long term, the more likely solution, is to introduce a new call called
RedisModule_GetClientFlags(), so that a command can detect if the caller
is a Lua script, and return an error, or avoid blocking at all.

Being the blocking API experimental right now, more work is needed in
this regard in order to reach a level well blocking module commands and
all the other Redis subsystems interact peacefully.

Now the effect is like the following:

    127.0.0.1:6379> eval "redis.call('hello.block',1,5000)" 0
    (error) ERR Error running script (call to
    f_b5ba35ff97bc1ef23debc4d6e9fd802da187ed53): @user_script:1: ERR
    Blocking module command called from Lua script

This commit fixes issue #4127 in the short term.
2017-07-23 12:55:37 +02:00
antirez 70f29030ae Fix typo in unblockClientFromModule() top comment. 2017-07-23 12:41:26 +02:00
antirez 01f1eb8f87 Make representClusterNodeFlags() more robust.
This function failed when an internal-only flag was set as an only flag
in a node: the string was trimmed expecting a final comma before
exiting the function, causing a crash. See issue #4142.
Moreover generation of flags representation only needed at DEBUG log
level was always performed: a waste of CPU time. This is fixed as well
by this commit.
2017-07-20 15:17:35 +02:00
antirez a19700091a Fix two bugs in moduleTypeLookupModuleByID().
The function cache was not working at all, and the function returned
wrong values if there where two or more modules exporting native data
types.

See issue #4131 for more details.
2017-07-20 14:59:42 +02:00
Leon Chen f32a44951a fix return wrong value of clusterDelNodeSlots 2017-07-20 17:24:38 +08:00
Leon Chen 9351639fe5 fix mismatch argument 2017-07-18 02:28:24 -05:00
WuYunlong e43fafda62 fix rewrite config: auto-aof-rewrite-min-size 2017-07-15 10:20:56 +08:00
Chris Lamb 009a48d144 Correct proceding -> proceeding typo. 2017-07-14 22:53:14 +01:00
antirez 0a18ba8719 changelog.tcl: output 100 lines. 2017-07-14 13:04:37 +02:00
antirez b36074dae3 Modules: fix thread safe context DB selection.
Before this fix the DB currenty selected by the client blocked was not
respected and operations were always performed on DB 0.
2017-07-14 13:02:15 +02:00
antirez 1ac18c801b Allow certain modules APIs only defining REDISMODULE_EXPERIMENTAL_API.
Those calls may be subject to changes in the future, so the user should
acknowledge it is using non stable API.
2017-07-14 12:07:52 +02:00
antirez 6fd40fdbc3 Modules documentation removed from source.
Moving to redis-doc repository to publish via Redis.io.
2017-07-14 11:33:59 +02:00
antirez f6d871f4c4 Markdown generation of Redis Modules API reference improved. 2017-07-14 11:29:31 +02:00
antirez 66c47a4d06 Fix replication of SLAVEOF inside transaction.
In Redis 4.0 replication, with the introduction of PSYNC2, masters and
slaves replicate commands to cascading slaves and to the replication
backlog itself in a different way compared to the past.

Masters actually replicate the effects of client commands.
Slaves just propagate what they receive from masters.

This mechanism can cause problems when the configuration of an instance
is changed from master to slave inside a transaction. For instance
we could send to a master instance the following sequence:

    MULTI
    SLAVEOF 127.0.0.1 0
    EXEC
    SLAVEOF NO ONE

Before the fixes in this commit, the MULTI command used to be propagated
into the replication backlog, however after the SLAVEOF command the
instance is a slave, so the EXEC implementation failed to also propagate
the EXEC command. When the slaves of the above instance reconnected,
they were incrementally synchronized just sending a "MULTI". This put
the master client (in the slaves) into MULTI state, breaking the
replication.

Notably even Redis Sentinel uses the above approach in order to guarantee
that configuration changes are always performed together with rewrites
of the configuration and with clients disconnection. Sentiel does:

    MULTI
    SLAVEOF ...
    CONFIG REWRITE
    CLIENT KILL TYPE normal
    EXEC

So this was a really problematic issue. However even with the fix in
this commit, that will add the final EXEC to the replication stream in
case the instance was switched from master to slave during the
transaction, the result would be to increment the slave replication
offset, so a successive reconnection with the new master, will not
permit a successful partial resynchronization: no way the new master can
provide us with the backlog needed, we incremented our offset to a value
that the new master cannot have.

However the EXEC implementation waits to emit the MULTI, so that if the
commands inside the transaction actually do not need to be replicated,
no commands propagation happens at all. From multi.c:

    if (!must_propagate && !(c->cmd->flags & (CMD_READONLY|CMD_ADMIN))) {
	execCommandPropagateMulti(c);
	must_propagate = 1;
    }

The above code is already modified by this commit you are reading.
Now also ADMIN commands do not trigger the emission of MULTI. It is actually
not clear why we do not just check for CMD_WRITE... Probably I wrote it this
way in order to make the code more reliable: better to over-emit MULTI
than not emitting it in time.

So this commit should indeed fix issue #3836 (verified), however it looks
like some reconsideration of this code path is needed in the long term.

BONUS POINT: The reverse bug.

Even in a read only slave "B", in a replication setup like:

	A -> B -> C

There are commands without the READONLY nor the ADMIN flag, that are also
not flagged as WRITE commands. An example is just the PING command.

So if we send B the following sequence:

    MULTI
    PING
    SLAVEOF NO ONE
    EXEC

The result will be the reverse bug, where only EXEC is emitted, but not the
previous MULTI. However this apparently does not create problems in practice
but it is yet another acknowledge of the fact some work is needed here
in order to make this code path less surprising.

Note that there are many different approaches we could follow. For instance
MULTI/EXEC blocks containing administrative commands may be allowed ONLY
if all the commands are administrative ones, otherwise they could be
denined. When allowed, the commands could simply never be replicated at all.
2017-07-12 11:07:28 +02:00
antirez e1b9781bda CLUSTER GETKEYSINSLOT: avoid overallocating.
Close #3911.
2017-07-11 15:49:09 +02:00
antirez 647406c1c1 Fix isHLLObjectOrReply() to handle integer encoded strings.
Close #3766.
2017-07-11 12:44:59 +02:00
antirez 89508a4fd4 Clients blocked in modules: free argv/argc later.
See issue #3844 for more information.
2017-07-11 12:33:01 +02:00
antirez f1308fcb08 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2017-07-11 09:46:58 +02:00
antirez ff1b4ccbca Event loop: call after sleep() only from top level.
In general we do not want before/after sleep() callbacks to be called
when we re-enter the event loop, since those calls are only designed in
order to perform operations every main iteration of the event loop, and
re-entering is often just a way to incrementally serve clietns with
error messages or other auxiliary operations. However, if we call the
callbacks, we are then forced to think at before/after sleep callbacks
as re-entrant, which is much harder without any good need.

However here there was also a clear bug: beforeSleep() was actually
never called when re-entering the event loop. But the new afterSleep()
callback was. This is broken and in this instance re-entering
afterSleep() caused a modules GIL dead lock.
2017-07-11 00:13:52 +02:00
Salvatore Sanfilippo cefcc33c41 Merge pull request #4113 from guybe7/module_io_bytes
Modules: Fix io->bytes calculation in RDB save
2017-07-10 19:14:34 +02:00
antirez a5cb21177a redis-check-aof: tell users there is a --fix option. 2017-07-10 16:41:25 +02:00
Guy Benoish cd3b6c9d5c Modules: Fix io->bytes calculation in RDB save 2017-07-10 14:41:57 +03:00
antirez 63ec3e0170 AOF check utility: ability to check files with RDB preamble. 2017-07-10 13:38:23 +02:00
Salvatore Sanfilippo ed5d0632b3 Merge pull request #3853 from itamarhaber/issue-3851
Sets up fake client to select current db in RM_Call()
2017-07-06 15:02:11 +02:00
Salvatore Sanfilippo 873fff969e Merge pull request #4105 from spinlock/unstable-networking
Optimize addReplyBulkSds for better performance
2017-07-06 14:31:08 +02:00
Salvatore Sanfilippo 8b7342cc67 Merge pull request #4106 from petersunbag/unstable
minor fix in listJoin().
2017-07-06 14:29:37 +02:00
sunweinan 16b407a1ff minor fix in listJoin(). 2017-07-06 19:47:21 +08:00
antirez cb3790a209 Free IO context if any in RDB loading code.
Thanks to @oranagra for spotting this bug.
2017-07-06 11:20:49 +02:00
antirez ed93fb8a29 Modules: DEBUG DIGEST interface. 2017-07-06 11:04:46 +02:00
spinlock db56f485a8 update Makefile for test-sds 2017-07-05 14:32:09 +00:00
spinlock b7b3e80a73 Optimize addReplyBulkSds for better performance 2017-07-05 14:25:05 +00:00
antirez ed7cbd5a4b Avoid closing invalid FDs to make Valgrind happier. 2017-07-05 15:40:25 +02:00
antirez fe48716c0c Modules: no MULTI/EXEC for commands replicated from async contexts.
They are technically like commands executed from external clients one
after the other, and do not constitute a single atomic entity.
2017-07-05 10:10:20 +02:00
Salvatore Sanfilippo 92863ae784 Merge pull request #4101 from dvirsky/fix_modules_reply_len
Proposed fix to #4100
2017-07-04 12:01:51 +02:00
antirez 80f2d39f64 Add symmetrical assertion to track c->reply_buffer infinite growth.
Redis clients need to have an instantaneous idea of the amount of memory
they are consuming (if the number is not exact should at least be
proportional to the actual memory usage). We do that adding and
subtracting the SDS length when pushing / popping from the client->reply
list. However it is quite simple to add bugs in such a setup, by not
taking the objects in the list and the count in sync. For such reason,
Redis has an assertion to track counts near 2^64: those are always the
result of the counter wrapping around because we subtract more than we
add. This commit adds the symmetrical assertion: when the list is empty
since we sent everything, the reply_bytes count should be zero. Thanks
to the new assertion it should be simple to also detect the other
problem, where the count slowly increases because of over-counting.
The assertion adds a conditional in the code that sends the buffer to
the socket but should not create any measurable performance slowdown,
listLength() just accesses a structure field, and this code path is
totally dominated by write(2).

Related to #4100.
2017-07-04 11:55:05 +02:00
Dvir Volk 4291a39afe fixed #4100 2017-07-04 00:02:19 +03:00
antirez b525305f9d Fix GEORADIUS edge case with huge radius.
This commit closes issue #3698, at least for now, since the root cause
was not fixed: the bounding box function, for huge radiuses, does not
return a correct bounding box, there are points still within the radius
that are left outside.

So when using GEORADIUS queries with radiuses in the order of 5000 km or
more, it was possible to see, at the edge of the area, certain points
not correctly reported.

Because the bounding box for now was used just as an optimization, and
such huge radiuses are not common, for now the optimization is just
switched off when the radius is near such magnitude.

Three test cases found by the Continuous Integration test were added, so
that we can easily trigger the bug again, both for regression testing
and in order to properly fix it as some point in the future.
2017-07-03 19:38:31 +02:00
Jun He 009bf5476a Fixed stack trace generation on aarch64
Change-Id: I9801239c98cb7362ed07e8b9ec2ba7e45749dba7
Signed-off-by: Jun He <jun.he@arm.com>
2017-07-03 08:47:55 +00:00
antirez 10d8d28164 redis-cli --latency: ability to run non interactively.
This feature was proposed by @rosmo in PR #2643 and later redesigned
in order to fit better with the other options for non-interactive modes
of redis-cli. The idea is basically to allow to collect latency
information in scripts, cron jobs or whateever, just running for a
limited time and then producing a single output.
2017-06-30 15:41:58 +02:00
antirez 501c7a725d Fix abort typo in Lua debugger help screen. 2017-06-30 12:12:00 +02:00
antirez 83bf475551 Added GEORADIUS(BYMEMBER)_RO variants for read-only operations.
Issue #4084 shows how for a design error, GEORADIUS is a write command
because of the STORE option. Because of this it does not work
on readonly slaves, gets redirected to masters in Redis Cluster even
when the connection is in READONLY mode and so forth.

To break backward compatibility at this stage, with Redis 4.0 to be in
advanced RC state, is problematic for the user base. The API can be
fixed into the unstable branch soon if we'll decide to do so in order to
be more consistent, and reease Redis 5.0 with this incompatibility in
the future. This is still unclear.

However, the ability to scale GEO queries in slaves easily is too
important so this commit adds two read-only variants to the GEORADIUS
and GEORADIUSBYMEMBER command: GEORADIUS_RO and GEORADIUSBYMEMBER_RO.
The commands are exactly as the original commands, but they do not
accept the STORE and STOREDIST options.
2017-06-30 10:03:37 +02:00
antirez 91a2ea769b HMSET and MSET implementations unified. HSET now variadic.
This is the first step towards getting rid of HMSET which is a command
that does not make much sense once HSET is variadic, and has a saner
return value.
2017-06-29 17:38:46 +02:00
Salvatore Sanfilippo 184b030535 Merge pull request #4075 from sgn1/brpop_keys
Fix Issues in blocking commands in cluster mode.
2017-06-27 17:51:19 +02:00
antirez 627713f627 RDB modules values serialization format version 2.
The original RDB serialization format was not parsable without the
module loaded, becuase the structure was managed only by the module
itself. Moreover RDB is a streaming protocol in the sense that it is
both produce di an append-only fashion, and is also sometimes directly
sent to the socket (in the case of diskless replication).

The fact that modules values cannot be parsed without the relevant
module loaded is a problem in many ways: RDB checking tools must have
loaded modules even for doing things not involving the value at all,
like splitting an RDB into N RDBs by key or alike, or just checking the
RDB for sanity.

In theory module values could be just a blob of data with a prefixed
length in order for us to be able to skip it. However prefixing the values
with a length would mean one of the following:

1. To be able to write some data at a previous offset. This breaks
stremaing.
2. To bufferize values before outputting them. This breaks performances.
3. To have some chunked RDB output format. This breaks simplicity.

Moreover, the above solution, still makes module values a totally opaque
matter, with the fowllowing problems:

1. The RDB check tool can just skip the value without being able to at
least check the general structure. For datasets composed mostly of
modules values this means to just check the outer level of the RDB not
actually doing any checko on most of the data itself.
2. It is not possible to do any recovering or processing of data for which a
module no longer exists in the future, or is unknown.

So this commit implements a different solution. The modules RDB
serialization API is composed if well defined calls to store integers,
floats, doubles or strings. After this commit, the parts generated by
the module API have a one-byte prefix for each of the above emitted
parts, and there is a final EOF byte as well. So even if we don't know
exactly how to interpret a module value, we can always parse it at an
high level, check the overall structure, understand the types used to
store the information, and easily skip the whole value.

The change is backward compatible: older RDB files can be still loaded
since the new encoding has a new RDB type: MODULE_2 (of value 7).
The commit also implements the ability to check RDB files for sanity
taking advantage of the new feature.
2017-06-27 13:19:16 +02:00
antirez e2a0d20737 ARM: Fix stack trace generation on crash. 2017-06-26 10:36:16 +02:00
antirez a771a41a56 Issue #4027: unify comment and modify return value in freeMemoryIfNeeded().
It looks safer to return C_OK from freeMemoryIfNeeded() when clients are
paused because returning C_ERR may prevent success of writes. It is
possible that there is no difference in practice since clients cannot
execute writes while clients are paused, but it looks more correct this
way, at least conceptually.

Related to PR #4028.
2017-06-23 11:42:25 +02:00
Salvatore Sanfilippo 2035627d34 Merge pull request #4028 from zintrepid/prevent_expirations_while_paused
Prevent expirations and evictions while paused
2017-06-23 11:39:02 +02:00
Suraj Narkhede 3e42887310 Fix following issues in blocking commands:
1. brpop last key index, thus checking all keys for slots.
2. Memory leak in clusterRedirectBlockedClientIfNeeded.
3. Remove while loop in clusterRedirectBlockedClientIfNeeded.
2017-06-23 00:30:21 -07:00
Suraj Narkhede a1149c6326 Fix brpop command table entry and redirect blocked clients. 2017-06-22 23:52:00 -07:00
antirez 7ef759e068 Aesthetic changes to #4068 PR to conform to Redis coding standard.
1. Inline if ... statement if short.
2. No lines over 80 columns.
2017-06-22 11:00:34 +02:00
Salvatore Sanfilippo 81da5ef3f9 Merge pull request #4068 from FreedomU007/unstable
Fix set with ex/px option when propagated to aof
2017-06-22 10:46:58 +02:00
xuzhou efed56d73a Optimize set command with ex/px when updating aof. 2017-06-22 11:06:40 +08:00
Salvatore Sanfilippo 5b631556be Merge pull request #3802 from flowly/bugfix-calc-stat-net-output-bytes
Bugfix calc stat net output bytes
2017-06-20 17:01:16 +02:00
Salvatore Sanfilippo 133f56d4f6 Merge pull request #4056 from season89/unstable
Fixed comments of slowlog duration
2017-06-20 16:55:29 +02:00
Salvatore Sanfilippo b0d58b5ddf Merge pull request #3659 from cbgbt/cli-elapsed
cli: Only print elapsed time on OUTPUT_STANDARD.
2017-06-20 16:53:56 +02:00
Salvatore Sanfilippo 6213929455 Merge pull request #4062 from concreted/patch-1
(fix) Update create-cluster README
2017-06-20 16:41:10 +02:00
antirez d78d6b22c6 redis-benchmark: add -t hset target. 2017-06-19 09:41:11 +02:00
Aric Huang 98d3075f2f (fix) Update create-cluster README
Fix a few typos/adjust wording in `create-cluster` README
2017-06-16 16:10:00 -07:00
xuzhou 809a73be97 Fix set with ex/px option when propagated to aof 2017-06-16 17:51:38 +08:00
antirez 6b1c3f89ab SLOWLOG: log offending client address and name. 2017-06-15 12:57:54 +02:00
antirez 02c7600ac7 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2017-06-14 18:29:53 +02:00
Qu Chen 29122cfa05 Implement getKeys procedure for georadius and georadiusbymember
commands.
2017-06-14 18:15:48 +02:00
xuchengxuan 44143324d0 Fixed comments of slowlog duration 2017-06-14 16:42:21 +08:00
Salvatore Sanfilippo d48581ab72 Merge pull request #4034 from amallia/patch-1
Fixed comment in clusterMsg version field
2017-06-13 06:28:23 -07:00
Salvatore Sanfilippo 617941ccf9 Merge pull request #4035 from amallia/patch-2
Removed duplicate 'sys/socket.h'  include
2017-06-13 06:27:31 -07:00
antirez 8606449fe1 Fix PERSIST expired key resuscitation issue #4048. 2017-06-13 10:35:51 +02:00
Antonio Mallia 88423d5bfd Removed duplicate 'sys/socket.h' include 2017-06-04 15:26:53 +01:00
Antonio Mallia eb83bd77be Fixed comment in clusterMsg version field 2017-06-04 15:09:05 +01:00
Zachary Marquez 541e4de976 Prevent expirations and evictions while paused
Proposed fix to https://github.com/antirez/redis/issues/4027
2017-06-01 16:28:40 -05:00
antirez b47078f14a More informative -MISCONF error message. 2017-05-19 12:03:30 +02:00
antirez f506064136 Collect fork() timing info only if fork succeeded. 2017-05-19 11:10:36 +02:00
antirez 2e9c6cac74 redis-cli --bigkeys: show error when TYPE fails.
Close #3993.
2017-05-15 11:22:28 +02:00
antirez 6b2316b9aa Merge branch 'unstable' of github.com:/antirez/redis into unstable 2017-05-10 13:15:30 +02:00
antirez e6ae9c9bab Modules TSC: use atomic var for server.unixtime.
This avoids Helgrind complaining, but we are actually not using
atomicGet() to get the unixtime value for now: too many places where it
is used and given tha time_t is word-sized it should be safe in all the
archs we support as it is.

On the other hand, Helgrind, when Redis is compiled with "make helgrind"
in order to force the __sync macros, will detect the write in
updateCachedTime() as a read (because atomic functions are used) and
will not complain about races.

This commit also includes minor refactoring of mutex initializations and
a "helgrind" target in the Makefile.
2017-05-10 10:04:16 +02:00
antirez abbeead6fa atomicvar.h: show used API in INFO. Add macro to force __sync builtin.
The __sync builtin can be correctly detected by Helgrind so to force it
is useful for testing. The API in the INFO output can be useful for
debugging after problems are reported.
2017-05-10 09:33:49 +02:00
Guy Benoish c71a77ad16 Merge branch 'unstable' of https://github.com/antirez/redis into unstable 2017-05-09 18:42:32 +03:00
antirez 6364bb1a79 zmalloc.c: remove thread safe mode, it's the default way. 2017-05-09 16:59:51 +02:00
antirez 61eb08813b Modules TSC: Add mutex for server.lruclock.
Only useful for when no atomic builtins are available.
2017-05-09 16:32:49 +02:00
antirez 42948bc052 Modules TSC: Improve inter-thread synchronization.
More work to do with server.unixtime and similar. Need to write Helgrind
suppression file in order to suppress the valse positives.
2017-05-09 11:57:09 +02:00
antirez ac4a9384e9 Simplify atomicvar.h usage by having the mutex name implicit. 2017-05-04 17:01:00 +02:00
antirez 71e9725dc7 Lazyfree: fix lazyfreeGetPendingObjectsCount() race reading counter. 2017-05-04 10:35:40 +02:00
antirez 86274d062f Modules TSC: HELLO.KEYS reply format fixed. 2017-05-03 23:43:49 +02:00
antirez 98c48456e8 Modules TSC: put the client in the pending write list. 2017-05-03 14:54:48 +02:00
antirez 833ebe4ab9 adlist: fix final list count in listJoin(). 2017-05-03 14:54:14 +02:00
antirez 8871307574 adlist: fix listJoin() to handle empty lists. 2017-05-03 14:15:25 +02:00
antirez b6e4ca1402 Modules: remove unused var in example module. 2017-05-03 14:10:21 +02:00
antirez e89b3655b8 Modules TSC: HELLO.KEYS example draft finished. 2017-05-03 14:08:12 +02:00
antirez a61003ccff Module: fix RedisModule_Call() "l" specifier to create a raw string. 2017-05-03 14:07:10 +02:00
antirez 441c323498 Modules TSC: Release the GIL for all the time we are blocked.
Instead of giving the module background operations just a small time to
run in the beforeSleep() function, we can have the lock released for all
the time we are blocked in the multiplexing syscall.
2017-05-03 11:26:21 +02:00
antirez 346677a10c Modules TSC: Export symbols of the new API. 2017-05-02 15:19:28 +02:00
antirez 5937533704 Modules TSC: Handling of RM_Reply* functions. 2017-05-02 15:05:39 +02:00
antirez 1617720681 Modules TSC: Basic TS context creeation and handling. 2017-05-02 12:53:10 +02:00
antirez 2d1ae6f06d Modules TSC: GIL and cooperative multi tasking setup. 2017-04-28 18:41:10 +02:00
antirez 78cfb8474b Regression test for #3899 fixed. 2017-04-28 11:16:39 +02:00
antirez 167765ed32 Regression test for PSYNC2 issue #3899 added.
Experimentally verified that it can trigger the issue reverting the fix.
At least on my system... Being the bug time/backlog dependant, it is
very hard to tell if this test will be able to trigger the problem
consistently, however even if it triggers the problem once in a while,
we'll see it in the CI environment at http://ci.redis.io.
2017-04-28 10:37:07 +02:00
antirez 0ee4c2743f PSYNC2: fix master cleanup when caching it.
The master client cleanup was incomplete: resetClient() was missing and
the output buffer of the client was not reset, so pending commands
related to the previous connection could be still sent.

The first problem caused the client argument vector to be, at times,
half populated, so that when the correct replication stream arrived the
protcol got mixed to the arugments creating invalid commands that nobody
called.

Thanks to @yangsiran for also investigating this problem, after
already providing important design / implementation hints for the
original PSYNC2 issues (see referenced Github issue).

Note that this commit adds a new function to the list library of Redis
in order to be able to reset a list without destroying it.

Related to issue #3899.
2017-04-27 17:08:37 +02:00
antirez 86b0650a4d Defrag: test currently disabled, too many false positives.
Related to #3786.
2017-04-22 15:59:57 +02:00
antirez 72dbb1d095 Defrag: fix test false positive.
Apparently 1.4 is too low compared to what you get in certain setups
(including mine). I raised it to 1.55 that hopefully is still enough to
test that the fragmentation went down from 1.7 but without incurring in
issues, however the test setup may be still fragile so certain times this
may lead to false positives again, it's hard to test for these things
in a determinsitic way.

Related to #3786.
2017-04-22 13:21:41 +02:00
oranagra f07d53f783 add test for active defrag 2017-04-22 13:17:09 +02:00
antirez d2ed982988 Revert "Jemalloc updated to 4.4.0."
This reverts commit 36c1acc222d29e6e2dc9fc25362e4faa471111bd.
2017-04-22 13:17:07 +02:00
antirez cb86d8916a Check event loop creation return value. Fix #3951.
Normally we never check for OOM conditions inside Redis since the
allocator will always return a pointer or abort the program on OOM
conditons. However we cannot have control on epool_create(), that may
fail for kernel OOM (according to the manual page) even if all the
parameters are correct, so the function aeCreateEventLoop() may indeed
return NULL and this condition must be checked.
2017-04-21 16:27:38 +02:00
Salvatore Sanfilippo 49a4f03692 Merge pull request #3950 from kensou97/unstable
update block->free after some diff data are written to the child process
2017-04-20 07:55:51 +02:00
antirez 29bc56e589 Fix getKeysUsingCommandTable() in cluster mode.
Close #3940.
2017-04-19 16:17:08 +02:00
antirez 62ef6778c9 PSYNC2: discard pending transactions from cached master.
During the review of the fix for #3899, @yangsiran identified an
implementation bug: given that the offset is now relative to the applied
part of the replication log, when we cache a master, the successive
PSYNC2 request will be made in order to *include* the transaction that
was not completely processed. This means that we need to discard any
pending transaction from our replication buffer: it will be re-executed.
2017-04-19 14:02:52 +02:00
antirez ad56461850 Fix PSYNC2 incomplete command bug as described in #3899.
This bug was discovered by @kevinmcgehee and constituted a major hidden
bug in the PSYNC2 implementation, caused by the propagation from the
master of incomplete commands to slaves.

The bug had several results:

1. Borrowing from Kevin text in the issue: "Given that slaves blindly
copy over their master's input into their own replication backlog over
successive read syscalls, it's possible that with large commands or
small TCP buffers, partial commands are present in this buffer. If the
master were to fail before successfully propagating the entire command
to a slave, the slaves will never execute the partial command (since the
client is invalidated) but will copy it to replication backlog which may
relay those invalid bytes to its slaves on PSYNC2, corrupting the
backlog and possibly other valid commands that follow the failover.
Simple command boundaries aren't sufficient to capture this, either,
because in the case of a MULTI/EXEC block, if the master successfully
propagates a subset of the commands but not the EXEC, then the
transaction in the backlog becomes corrupt and could corrupt other
slaves that consume this data."

2. As identified by @yangsiran later, there is another effect of the
bug. For the same mechanism of the first problem, a slave having another
slave, could receive a full resynchronization request with an already
half-applied command in the backlog. Once the RDB is ready, it will be
sent to the slave, and the replication will continue sending to the
sub-slave the other half of the command, which is not valid.

The fix, designed by @yangsiran and @antirez, and implemented by
@antirez, uses a secondary buffer in order to feed the sub-masters and
update the replication backlog and offsets, only when a given part of
the query buffer is actually *applied* to the state of the instance,
that is, when the command gets processed and the command is not pending
in the Redis transaction buffer because of CLIENT_MULTI state.

Given that now the backlog and offsets representation are in agreement
with the actual processed commands, both issue 1 and 2 should no longer
be possible.

Thanks to @kevinmcgehee, @yangsiran and @oranagra for their work in
identifying and designing a fix for this problem.
2017-04-19 10:25:45 +02:00
Salvatore Sanfilippo be2a34d871 Merge pull request #3945 from badboy/dicthash-bench-compile
Reorder to make dict-benchmark compile on Linux
2017-04-18 16:31:18 +02:00
antirez f52c65edf2 Fix #3848 by closing the descriptor on error. 2017-04-18 16:24:06 +02:00
antirez e3e243744c Merge branch 'unstable' of github.com:/antirez/redis into unstable 2017-04-18 16:15:24 +02:00
antirez 199b0b0c36 Fix descriptor leak. Close #3848. 2017-04-18 16:15:16 +02:00
Salvatore Sanfilippo d840bc0dbd Merge pull request #3856 from viennadd/issue-3847
fix #3847: add close socket before return ANET_ERR.
2017-04-18 16:13:23 +02:00
张文康 f34c984f2f update block->free after some diff data are written to the child process 2017-04-18 20:10:08 +08:00
antirez 083636b801 Clarify why we save ziplist elements in revserse order.
Also get rid of variables that are now kinda redundant, since the
dictionary iterator was removed.

This is related to PR #3949.
2017-04-18 11:01:47 +02:00
Salvatore Sanfilippo 174c5846a5 Merge pull request #3949 from spinlock/unstable-rdb-encoding
rdb: saving skiplist in reversed order to accelerate the deserialisation process
2017-04-18 10:56:57 +02:00
Jan-Erik Rediger 606c74baf6 Reorder to make dict-benchmark compile on Linux
Fixes #3944
2017-04-17 13:37:59 +02:00
spinlock ed5d5d6633 rdb: saving skiplist in reversed order to accelerate the deserialisation process 2017-04-17 13:22:34 +08:00
antirez 4d4f22843c Cluster: discard pong times in the future.
However we allow for 500 milliseconds of tolerance, in order to
avoid often discarding semantically valid info (the node is up)
because of natural few milliseconds desync among servers even when
NTP is used.

Note that anyway we should ping the node from time to time regardless and
discover if it's actually down from our point of view, since no update
is accepted while we have an active ping on the node.

Related to #3929.
2017-04-15 10:12:08 +02:00
antirez a2e3dc0a1d Test: fix, hopefully, false PSYNC failure like in issue #2715.
And many other related Github issues... all reporting the same problem.
There was probably just not enough backlog in certain unlucky runs.
I'll ask people that can reporduce if they see now this as fixed as
well.
2017-04-14 17:53:11 +02:00
antirez 0530c7e564 Cluster: always add PFAIL nodes at end of gossip section.
To rely on the fact that nodes in PFAIL state will be shared around by
randomly adding them in the gossip section is a weak assumption,
especially after changes related to sending less ping/pong packets.

We want to always include gossip entries for all the nodes that are in
PFAIL state, so that the PFAIL -> FAIL state promotion can happen much
faster and reliably.

Related to #3929.
2017-04-14 13:39:49 +02:00
antirez 06de4577d6 Cluster: fix gossip section ping/pong times encoding.
The gossip section times are 32 bit, so cannot store the milliseconds
time but just the seconds approximation, which is good enough for our
uses. At the same time however, when comparing the gossip section times
of other nodes with our node's view, we need to convert back to
milliseconds.

Related to #3929. Without this change the patch to reduce the traffic in
the bus message does not work.
2017-04-14 11:01:22 +02:00
antirez 81a36f41ec Cluster: add clean-logs command to create-cluster script. 2017-04-14 10:52:00 +02:00
antirez de2eed4838 Cluster: decrease ping/pong traffic by trusting other nodes reports.
Cluster of bigger sizes tend to have a lot of traffic in the cluster bus
just for failure detection: a node will try to get a ping reply from
another node no longer than when the half the node timeout would elapsed,
in order to avoid a false positive.

However this means that if we have N nodes and the node timeout is set
to, for instance M seconds, we'll have to ping N nodes every M/2
seconds. This N*M/2 pings will receive the same number of pongs, so
a total of N*M packets per node. However given that we have a total of N
nodes doing this, the total number of messages will be N*N*M.

In a 100 nodes cluster with a timeout of 60 seconds, this translates
to a total of 100*100*30 packets per second, summing all the packets
exchanged by all the nodes.

This is, as you can guess, a lot... So this patch changes the
implementation in a very simple way in order to trust the reports of
other nodes: if a node A reports a node B as alive at least up to
a given time, we update our view accordingly.

The problem with this approach is that it could result into a subset of
nodes being able to reach a given node X, and preventing others from
detecting that is actually not reachable from the majority of nodes.
So the above algorithm is refined by trusting other nodes only if we do
not have currently a ping pending for the node X, and if there are no
failure reports for that node.

Since each node, anyway, pings 10 other nodes every second (one node
every 100 milliseconds), anyway eventually even trusting the other nodes
reports, we will detect if a given node is down from our POV.

Now to understand the number of packets that the cluster would exchange
for failure detection with the patch, we can start considering the
random PINGs that the cluster sent anyway as base line:
Each node sends 10 packets per second, so the total traffic if no
additioal packets would be sent, including PONG packets, would be:

    Total messages per second = N*10*2

However by trusting other nodes gossip sections will not AWALYS prevent
pinging nodes for the "half timeout reached" rule all the times. The
math involved in computing the actual rate as N and M change is quite
complex and depends also on another parameter, which is the number of
entries in the gossip section of PING and PONG packets. However it is
possible to compare what happens in cluster of different sizes
experimentally. After applying this patch a very important reduction in
the number of packets exchanged is trivial to observe, without apparent
impacts on the failure detection performances.

Actual numbers with different cluster sizes should be published in the
Reids Cluster documentation in the future.

Related to #3929.
2017-04-14 10:43:53 +02:00
antirez c628ae114b Cluster: collect more specific bus messages stats.
First step in order to change Cluster in order to use less messages.
Related to issue #3929.
2017-04-13 19:22:35 +02:00
Itamar Haber cac5a8b65d Changes command stats iteration to being dict-based
With the addition of modules, looping over the redisCommandTable
misses any added commands. By moving to dictionary iteration this
is resolved.
2017-04-13 17:03:46 +03:00
antirez 341adb516b Fix typo in feedReplicationBacklog() top comment. 2017-04-12 12:28:05 +02:00
antirez 6081a5873c Add a top comment in crucial functions inside networking.c. 2017-04-12 10:12:27 +02:00
antirez 7c415014b0 Set lua-time-limit default value at safe place.
Otherwise, as it was, it will overwrite whatever the user set.

Close #3703.
2017-04-11 16:56:00 +02:00
antirez ffce9ebf9b Fix preprocessor if/else chain broken in order to fix #3927. 2017-04-11 16:54:27 +02:00
antirez de3e670282 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2017-04-11 16:45:49 +02:00
antirez 7e0c3177d6 Fix zmalloc_get_memory_size() ifdefs to actually use the else branch.
Close #3927.
2017-04-11 16:45:11 +02:00
Salvatore Sanfilippo 3b0a442dd2 Merge pull request #3924 from lorneli/unstable
Expire: Update comment of activeExpireCycle function
2017-04-11 16:31:55 +02:00
antirez 71c350f73b Make more obvious why there was issue #3843. 2017-04-10 13:17:05 +02:00
Salvatore Sanfilippo cccd21d5c6 Merge pull request #3843 from dvirsky/fix_bc_free
fixed free of blocked client before refering to it
2017-04-10 13:14:52 +02:00
antirez 60ffbb72b4 Fix modules blocking commands awake delay.
If a thread unblocks a client blocked in a module command, by using the
RedisMdoule_UnblockClient() API, the event loop may not be awaken until
the next timeout of the multiplexing API or the next unrelated I/O
operation on other clients. We actually want the client to be served
ASAP, so a mechanism is needed in order for the unblocking API to inform
Redis that there is a client to serve ASAP.

This commit fixes the issue using the old trick of the pipe: when a
client needs to be unblocked, a byte is written in a pipe. When we run
the list of clients blocked in modules, we consume all the bytes
written in the pipe. Writes and reads are performed inside the context
of the mutex, so no race is possible in which we consume the bytes that
are actually related to an awake request for a client that should still
be put into the list of clients to unblock.

It was verified that after the fix the server handles the blocked
clients with the expected short delay.

Thanks to @dvirsky for understanding there was such a problem and
reporting it.
2017-04-10 09:33:21 +02:00
antirez f9db3144d6 Rax library updated.
Important bugs fixed.
2017-04-08 17:31:13 +02:00
lorneli e8b44eb33a Expire: Update comment of activeExpireCycle function
The macro REDIS_EXPIRELOOKUPS_TIME_PERC has been replaced by
ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC in commit
6500fabfb8.
2017-04-08 15:15:24 +08:00
antirez fcad87788a Rax library updated. 2017-04-07 08:46:39 +02:00
antirez de52b6375b Cluster: hash slots tracking using a radix tree. 2017-03-27 16:37:22 +02:00
Salvatore Sanfilippo 7d036aadc2 Merge pull request #3875 from oranagra/lfu_tests
add LFU policies to the test suite, just for coverage
2017-03-15 09:18:04 +01:00
Oran Agra 499595f510 add LFU policies to the test suite, just for coverage 2017-03-15 01:05:15 -07:00
antirez 01f56d44dc Use sha256 instead of sha1 to generate tarball hashes. 2017-03-09 13:49:36 +01:00
vienna bcb1240ccf fix #3847: add close socket before return ANET_ERR. 2017-03-07 16:14:05 +00:00
itamar a04ba58d9a Sets up fake client to select current db in RM_Call() 2017-03-06 14:37:10 +02:00
Guy Benoish dcbf01295b Merge branch 'unstable' of https://github.com/antirez/redis into unstable 2017-03-02 13:25:05 +02:00
Dvir Volk c40322945a fixed free of blocked client before refering to it 2017-03-01 16:51:01 +02:00
Salvatore Sanfilippo 1f0dae3c7f Makefile: fix building with Solaris C compiler, 64 bit. 2017-02-23 16:53:39 +01:00
antirez 64577122ec Merge branch 'sparc' of ssh://209.141.57.197:12222//export/home/antirez/redis into sparc 2017-02-23 15:35:01 +01:00
Salvatore Sanfilippo 7a4c9474f8 Use ARM unaligned accesses ifdefs for SPARC as well. 2017-02-23 22:39:44 +08:00
Salvatore Sanfilippo 08007ed442 Fix BITPOS unaligned memory access. 2017-02-23 22:38:44 +08:00
oranagra 8f9e82e868 when a slave experiances an error on commands that come from master, print to the log
since slave isn't replying to it's master, these errors go unnoticed.
since we don't expect the master to send garbadge to the slave, this should be safe.
(as long as we don't log OOM errors there)
2017-02-23 03:44:42 -08:00
oranagra ecda721a3e add SDS_NOINIT option to sdsnewlen to avoid unnecessary memsets.
this commit also contains small bugfix in rdbLoadLzfStringObject
a bug that currently has no implications.
2017-02-23 03:04:08 -08:00
antirez 9df5191d0a Solaris fixes about tail usage and atomic vars.
Testing with Solaris C compiler (SunOS 5.11 11.2 sun4v sparc sun4v)
there were issues compiling due to atomicvar.h and running the
tests also failed because of "tail" usage not conform with Solaris
tail implementation. This commit fixes both the issues.
2017-02-22 13:08:21 +01:00
antirez 384ae4032d Test: replication-psync, wait more to detect write load.
Slow systems like the original Raspberry PI need more time
than 5 seconds to start the script and detect writes.
After fixing the Raspberry PI can pass the unit without issues.
2017-02-22 12:27:01 +01:00
antirez d0e1affa70 Test: fix conditional execution of HINCRBYFLOAT representation test. 2017-02-22 12:00:09 +01:00
antirez 20a4c6cff3 Merge branch 'siphash' into unstable 2017-02-21 17:10:10 +01:00
antirez d4cf5babd2 Merge branch 'arm' into unstable 2017-02-21 17:10:06 +01:00
antirez afd60f3f74 SipHash 2-4 -> SipHash 1-2.
For performance reasons we use a reduced rounds variant of
SipHash. This should still provide enough protection and the
effects in the hash table distribution are non existing.
If some real world attack on SipHash 1-2 will be found we can
trivially switch to something more secure. Anyway it is a
big step forward from Murmurhash, for which it is trivial to
generate *seed independent* colliding keys... The speed
penatly introduced by SipHash 2-4, around 4%, was a too big
price to pay compared to the effectiveness of the HashDoS
attack against SipHash 1-2, and considering so far in the
Redis history, no such an incident ever happened even while
using trivially to collide hash functions.
2017-02-21 17:07:28 +01:00
antirez 5ed1b45aad freeMemoryIfNeeded(): improve code and lazyfree handling.
1. Refactor memory overhead computation into a function.
2. Every 10 keys evicted, check if memory usage already reached
   the target value directly, since we otherwise don't count all
   the memory reclaimed by the background thread right now.
2017-02-21 12:55:59 +01:00
antirez 49cb8be3fd Use locale agnostic tolower() in dict.c hash function. 2017-02-20 17:39:44 +01:00
antirez fc20100861 SipHash x86 optimizations. 2017-02-20 17:32:46 +01:00
antirez b49721d57d Use SipHash hash function to mitigate HashDos attempts.
This change attempts to switch to an hash function which mitigates
the effects of the HashDoS attack (denial of service attack trying
to force data structures to worst case behavior) while at the same time
providing Redis with an hash function that does not expect the input
data to be word aligned, a condition no longer true now that sds.c
strings have a varialbe length header.

Note that it is possible sometimes that even using an hash function
for which collisions cannot be generated without knowing the seed,
special implementation details or the exposure of the seed in an
indirect way (for example the ability to add elements to a Set and
check the return in which Redis returns them with SMEMBERS) may
make the attacker's life simpler in the process of trying to guess
the correct seed, however the next step would be to switch to a
log(N) data structure when too many items in a single bucket are
detected: this seems like an overkill in the case of Redis.

SPEED REGRESION TESTS:

In order to verify that switching from MurmurHash to SipHash had
no impact on speed, a set of benchmarks involving fast insertion
of 5 million of keys were performed.

The result shows Redis with SipHash in high pipelining conditions
to be about 4% slower compared to using the previous hash function.
However this could partially be related to the fact that the current
implementation does not attempt to hash whole words at a time but
reads single bytes, in order to have an output which is endian-netural
and at the same time working on systems where unaligned memory accesses
are a problem.

Further X86 specific optimizations should be tested, the function
may easily get at the same level of MurMurHash2 if a few optimizations
are performed.
2017-02-20 17:29:17 +01:00
John.Koepi 9383cda5ff fix #2883, #2857 pipe fds leak when fork() failed on bg aof rw 2017-02-20 10:22:57 +01:00
antirez c9f6868c51 Don't leak file descriptor on syncWithMaster().
Close #3804.
2017-02-20 10:18:41 +01:00
Salvatore Sanfilippo d921cbea3c ARM: Avoid fast path for BITOP.
GCC will produce certain unaligned multi load-store instructions
that will be trapped by the Linux kernel since ARM v6 cannot
handle them with unaligned addresses. Better to use the slower
but safer implementation instead of generating the exception which
should be anyway very slow.
2017-02-19 15:07:08 +00:00
Salvatore Sanfilippo bd6b031738 ARM: Use libc malloc by default.
I'm not sure how much test Jemalloc gets on ARM, moreover
compiling Redis with Jemalloc support in not very powerful
devices, like most ARMs people will build Redis on, is extremely
slow. It is possible to enable Jemalloc build anyway if needed
by using "make MALLOC=jemalloc".
2017-02-19 15:02:37 +00:00
Salvatore Sanfilippo d17abbf4e2 ARM: Avoid memcpy() in MurmurHash64A() if we are using 64 bit ARM.
However note that in architectures supporting 64 bit unaligned
accesses memcpy(...,...,8) is likely translated to a simple
word memory movement anyway.
2017-02-19 15:00:46 +00:00
Salvatore Sanfilippo 18b4a41899 ARM: Fix 64 bit unaligned access in MurmurHash64A(). 2017-02-19 14:01:58 +00:00
minghang.zmh a20c8bba4a fix server.stat_net_output_bytes calc bug 2017-02-10 20:13:01 +08:00
flowly 609ec3d4c2 Merge pull request #1 from antirez/unstable
update to upstream
2017-02-10 19:53:36 +08:00
antirez 38894de7f5 Fix MIGRATE closing of cached socket on error.
After investigating issue #3796, it was discovered that MIGRATE
could call migrateCloseSocket() after the original MIGRATE c->argv
was already rewritten as a DEL operation. As a result the host/port
passed to migrateCloseSocket() could be anything, often a NULL pointer
that gets deferenced crashing the server.

Now the socket is closed at an earlier time when there is a socket
error in a later stage where no retry will be performed, before we
rewrite the argument vector. Moreover a check was added so that later,
in the socket_err label, there is no further attempt at closing the
socket if the argument was rewritten.

This fix should resolve the bug reported in #3796.
2017-02-09 09:58:38 +01:00
antirez 34b3b01261 Fix ziplist fix... 2017-02-01 17:01:31 +01:00
antirez c750d3215e Ziplist: insertion bug under particular conditions fixed.
Ziplists had a bug that was discovered while investigating a different
issue, resulting in a corrupted ziplist representation, and a likely
segmentation foult and/or data corruption of the last element of the
ziplist, once the ziplist is accessed again.

The bug happens when a specific set of insertions / deletions is
performed so that an entry is encoded to have a "prevlen" field (the
length of the previous entry) of 5 bytes but with a count that could be
encoded in a "prevlen" field of a since byte. This could happen when the
"cascading update" process called by ziplistInsert()/ziplistDelete() in
certain contitious forces the prevlen to be bigger than necessary in
order to avoid too much data moving around.

Once such an entry is generated, inserting a very small entry
immediately before it will result in a resizing of the ziplist for a
count smaller than the current ziplist length (which is a violation,
inserting code expects the ziplist to get bigger actually). So an FF
byte is inserted in a misplaced position. Moreover a realloc() is
performed with a count smaller than the ziplist current length so the
final bytes could be trashed as well.

SECURITY IMPLICATIONS:

Currently it looks like an attacker can only crash a Redis server by
providing specifically choosen commands. However a FF byte is written
and there are other memory operations that depend on a wrong count, so
even if it is not immediately apparent how to mount an attack in order
to execute code remotely, it is not impossible at all that this could be
done. Attacks always get better... and we did not spent enough time in
order to think how to exploit this issue, but security researchers
or malicious attackers could.
2017-02-01 15:01:59 +01:00
antirez 5bd9578445 ziplist: better comments, some refactoring. 2017-01-30 10:12:47 +01:00
antirez 362c8f80b1 Jemalloc updated to 4.4.0.
The original jemalloc source tree was modified to:

1. Remove the configure error that prevents nested builds.
2. Insert the Redis private Jemalloc API in order to allow the
Redis fragmentation function to work.
2017-01-30 09:58:34 +01:00
Salvatore Sanfilippo 7dc57bd96a Merge pull request #3784 from badboy/dont-divide-by-zero
Don't divide by zero
2017-01-27 16:26:24 +01:00
Jan-Erik Rediger 978e62da15 Don't divide by zero
Previously Redis crashed on `MEMORY DOCTOR` when it has no slaves attached.

Fixes #3783
2017-01-27 16:24:14 +01:00
Salvatore Sanfilippo 9631ac63e1 Merge pull request #3778 from MiterV1/unstable
Change switch statment to if statment
2017-01-26 13:41:59 +01:00
miter 8b565df63c Change switch statment to if statment 2017-01-26 21:36:26 +09:00
Salvatore Sanfilippo 3c1963d513 Merge pull request #3657 from itamarhaber/patch-9
Verify pairs are provided after ZADD's subcommands
2017-01-25 09:31:47 +01:00
Salvatore Sanfilippo 2da38b624e Merge pull request #3712 from oranagra/fix_assert_debug_digest
fix rare assertion in DEBUG DIGEST
2017-01-20 11:01:43 +01:00
antirez d3516e69c7 Add panic() into redisassert.h.
This header file is for libs, like ziplist.c, that we want to leave
almost separted from the core. The panic() calls will be easy to delete
in order to use such files outside, but the debugging info we gain are
very valuable compared to simple assertions where it is not possible to
print debugging info.
2017-01-18 17:12:07 +01:00
antirez c6dfff5b61 serverPanic(): allow printf() alike formatting.
This is of great interest because allows us to print debugging
informations that could be of useful when debugging, like in the
following example:

    serverPanic("Unexpected encoding for object %d, %d",
        obj->type, obj->encoding);
2017-01-18 17:05:10 +01:00
antirez eee3784224 Ziplist: remove static from functions, they prevent good crash reports. 2017-01-13 11:55:13 +01:00
Salvatore Sanfilippo 644999645b Merge pull request #3734 from badboy/avoid-command
Initialize help only in repl mode
2017-01-13 11:32:22 +01:00
antirez c0837ddbcc Use const in modules types mem_usage method.
As suggested by @itamarhaber.
2017-01-12 12:47:46 +01:00
antirez 46f2263a45 Defrag: don't crash when a module value is encountered. 2017-01-12 09:50:40 +01:00
antirez d4593397a7 MEMORY USAGE: support for modules data types.
As a side effect of supporting it, we no longer crash when MEMORY USAGE
is called against a module data type.

Close #3637.
2017-01-12 09:47:57 +01:00
Guy Benoish fa59d52356 enlarged buffer given to ld2string 2017-01-11 19:24:19 +02:00
antirez 701a69ed4d Defrag: document the feature in redis.conf. 2017-01-11 17:24:49 +01:00
antirez eabcf0fed8 Defrag: not enabled by default. Error on CONFIG SET if not available. 2017-01-11 15:43:08 +01:00
antirez 57c80d1fca Defrag: fix function name typo defarg -> defrag. 2017-01-11 15:38:12 +01:00
antirez f11a12e915 Defrag: do not crash on empty quicklist. 2017-01-11 15:38:09 +01:00
antirez da55ed0032 Defrag: fix comments & code to conform to the Redis code base.
Don't go over 80 cols. Start with captial letter, capital letter afer
point, end comment with a point and so forth. No actual code behavior
touched at all.
2017-01-10 11:33:50 +01:00
antirez e4f25b015c Defrag: activate it only if running modified version of Jemalloc.
This commit also includes minor aesthetic changes like removal of
trailing spaces.
2017-01-10 11:25:39 +01:00
Jan-Erik Rediger 9c77672458 Initialize help only in repl mode 2017-01-08 18:29:22 +01:00
oranagra 763f49243d active defrag improvements 2017-01-02 09:42:32 +02:00
oranagra 53511a429c active memory defragmentation 2016-12-30 03:37:52 +02:00
oranagra 8528bbaf6c fix rare assertion in DEBUG DIGEST
getExpire calls dictFind which can do rehashing.
found by calling computeDatasetDigest from serverCron and running the test suite.
2016-12-24 17:27:58 +02:00
antirez 8155b2fca4 deps/hiredis updated to latest version.
Close #3687.
2016-12-21 12:12:36 +01:00
Salvatore Sanfilippo 44c6326c11 Merge pull request #3242 from whatacold/unstable
fix the wrong description of intsetGet().
2016-12-20 15:39:56 +01:00
Salvatore Sanfilippo 4a670f7453 Merge pull request #3498 from ZhukovAlexander/patch-1
Fix an article usage
2016-12-20 13:39:29 +01:00
antirez d13144285b Merge branch 'unstable' of github.com:/antirez/redis into unstable 2016-12-20 13:33:10 +01:00
Salvatore Sanfilippo c77e46af5c Merge pull request #3696 from jstncarvalho/FixMissingBrackets_ZIP_DECODE_LENGTH
Fix missing brackets around encoding variable in ZIP_DECODE_LENGTH macro
2016-12-20 13:32:54 +01:00
antirez 29fbdde45f Geo: fuzzy test inconsistency report fixed to show all points.
We need to report all the points added into the set, not just the ones
matching the Tcl distance algo.
2016-12-20 12:22:11 +01:00
antirez db1e1e63ce Geo: fix GEOHASH return value for consistency.
The same thing observed in #3551 by gnethercutt also fixed for
GEOHASH as the original PR did.
2016-12-20 10:20:13 +01:00
antirez ebab46b1fb Geo: fix edge case return values for uniformity.
There were two cases outlined in issue #3512 and PR #3551 where
the Geo API returned unexpected results: empty strings where NULL
replies were expected, or a single null reply where an array was
expected. This violates the Redis principle that Redis replies for
existing keys or elements should be indistinguishable.

This is technically an API breakage so will be merged only into 4.0 and
specified in the changelog in the list of breaking compatibilities, even
if it is not very likely that actual code will be affected, hopefully,
since with the past behavior basically there was to acconut for *both*
the possibilities, and the new behavior is always one of the two, but
in a consistent way.
2016-12-20 10:12:38 +01:00
Justin Carvalho 5aa18b3547 Fix missing brackets around encoding variable in ZIP_DECODE_LENGTH macro 2016-12-19 17:37:41 -05:00
antirez 1ffd2f315b Fix test "server is up" detection after logging changes. 2016-12-19 16:49:58 +01:00
antirez d2d95729fe Remove first version of ASCII wave, later discarded. 2016-12-19 16:45:18 +01:00
antirez 03f1cc2023 Only show Redis logo if logging to stdout / TTY.
You can still force the logo in the normal logs.
For motivations, check issue #3112. For me the reason is that actually
the logo is nice to have in interactive sessions, but inside the logs
kinda loses its usefulness, but for the ability of users to recognize
restarts easily: for this reason the new startup sequence shows a one
liner ASCII "wave" so that there is still a bit of visual clue.

Startup logging was modified in order to log events in more obvious
ways, and to log more events. Also certain important informations are
now more easy to parse/grep since they are printed in field=value style.

The option --always-show-logo in redis.conf was added, defaulting to no.
2016-12-19 16:41:47 +01:00
antirez 0af9df2aa5 adjustOpenFilesLimit() comment made hopefully more clear. 2016-12-19 08:53:29 +01:00
Salvatore Sanfilippo bac364fac3 Merge pull request #3603 from oranagra/adjustOpenFilesLimit_overflow
fix unsigned int overflow in adjustOpenFilesLimit
2016-12-19 08:48:44 +01:00
Salvatore Sanfilippo c8f102e5bb Merge pull request #3605 from hylepo/unstable
Fixing typo in the usage of redis-benchmark
2016-12-19 08:20:01 +01:00
Salvatore Sanfilippo 302b8a7ec2 Merge pull request #3643 from andyli028/unstable
Modify MIN->MAX
2016-12-19 08:19:10 +01:00
antirez 8bd0dbd805 Hopefully improve code comments for issue #3616.
This commit also contains other changes in order to conform the code to
the Redis core style, specifically 80 chars max per line, smart
conditionals in the same line:

    if (that) do_this();
2016-12-16 17:48:38 +01:00
Salvatore Sanfilippo 54cbdaf69e Merge pull request #3616 from oranagra/stop_aofrw_before_rdbload
CoW improvement, stop AOFRW before flushing and parsing slave RDB
2016-12-16 17:43:20 +01:00
Salvatore Sanfilippo fc86d34d6e Merge pull request #3661 from itamarhaber/module-doc2
Corrects a couple of omissions in the modules docs
2016-12-16 16:53:13 +01:00
antirez 6820b5cae1 Switch PFCOUNT to LogLog-Beta algorithm.
The new algorithm provides the same speed with a smaller error for
cardinalities in the range 0-100k. Before switching, the new and old
algorithm behavior was studied in details in the context of
issue #3677. You can find a few graphs and motivations there.
2016-12-16 11:07:30 +01:00
antirez be49812350 Use llroundl() before converting loglog-beta output to integer.
Otherwise for small cardinalities the algorithm will output something
like, for example, 4.99 for a candinality of 5, that will be converted
to 4 producing a huge error.
2016-12-16 11:07:30 +01:00
antirez c3c1c53df3 Fix HLL gnuplot graph generator script for new redis-rb versions.
The PFADD now takes an array and has mandatory two arguments.
2016-12-16 11:07:30 +01:00
Harish Murthy c1545d2f23 LogLog-Beta Algorithm support within HLL
Config option to use LogLog-Beta Algorithm for Cardinality
2016-12-16 11:07:30 +01:00
Salvatore Sanfilippo 16220de83f Merge pull request #3686 from dvirsky/fix_lowlevel_zrange
fixed stop condition in RM_ZsetRangeNext and RM_ZsetRangePrev
2016-12-16 09:20:47 +01:00
antirez 044068eb7f ziplist.c explanation of format improved a bit. 2016-12-16 09:04:57 +01:00
antirez d02963eec9 DEBUG: new "ziplist" subcommand added. Dumps a ziplist on stdout.
The commit improves ziplistRepr() and adds a new debugging subcommand so
that we can trigger the dump directly from the Redis API.
This command capability was used while investigating issue #3684.
2016-12-16 09:02:50 +01:00
Dvir Volk d8f4c209ab fixed stop condition in RM_ZsetRangeNext and RM_ZsetRangePrev 2016-12-15 00:07:20 +02:00
antirez 768220fcdc MIGRATE: Remove upfront ttl initialization.
After the fix for #3673 the ttl var is always initialized inside the
loop itself, so the early initialization is not needed.

Variables declaration also moved to a more local scope.
2016-12-14 12:43:55 +01:00
Salvatore Sanfilippo 28a29d6922 Merge pull request #3673 from badboy/reset-ttl-on-migrating
Reset the ttl for additional keys
2016-12-14 12:41:00 +01:00
antirez 403ed3368a Writable slaves expires: unit test. 2016-12-13 16:28:12 +01:00
antirez 03119a0831 Writable slaves expires: fix leak in key tracking.
We need to use a dictionary type that frees the key, since we copy the
keys in the dictionary we use to track expires created in the slave
side.
2016-12-13 16:27:13 +01:00
antirez 3f302b9ecf INFO: show num of slave-expires keys tracked. 2016-12-13 16:02:29 +01:00
antirez 1ae5a13266 Fix created->created typo in expire.c 2016-12-13 12:21:15 +01:00
antirez a8a74bb8a5 Replication: fix the infamous key leakage of writable slaves + EXPIRE.
BACKGROUND AND USE CASEj

Redis slaves are normally write only, however the supprot a "writable"
mode which is very handy when scaling reads on slaves, that actually
need write operations in order to access data. For instance imagine
having slaves replicating certain Sets keys from the master. When
accessing the data on the slave, we want to peform intersections between
such Sets values. However we don't want to intersect each time: to cache
the intersection for some time often is a good idea.

To do so, it is possible to setup a slave as a writable slave, and
perform the intersection on the slave side, perhaps setting a TTL on the
resulting key so that it will expire after some time.

THE BUG

Problem: in order to have a consistent replication, expiring of keys in
Redis replication is up to the master, that synthesize DEL operations to
send in the replication stream. However slaves logically expire keys
by hiding them from read attempts from clients so that if the master did
not promptly sent a DEL, the client still see logically expired keys
as non existing.

Because slaves don't actively expire keys by actually evicting them but
just masking from the POV of read operations, if a key is created in a
writable slave, and an expire is set, the key will be leaked forever:

1. No DEL will be received from the master, which does not know about
such a key at all.

2. No eviction will be performed by the slave, since it needs to disable
eviction because it's up to masters, otherwise consistency of data is
lost.

THE FIX

In order to fix the problem, the slave should be able to tag keys that
were created in the slave side and have an expire set in some way.

My solution involved using an unique additional dictionary created by
the writable slave only if needed. The dictionary is obviously keyed by
the key name that we need to track: all the keys that are set with an
expire directly by a client writing to the slave are tracked.

The value in the dictionary is a bitmap of all the DBs where such a key
name need to be tracked, so that we can use a single dictionary to track
keys in all the DBs used by the slave (actually this limits the solution
to the first 64 DBs, but the default with Redis is to use 16 DBs).

This solution allows to pay both a small complexity and CPU penalty,
which is zero when the feature is not used, actually. The slave-side
eviction is encapsulated in code which is not coupled with the rest of
the Redis core, if not for the hook to track the keys.

TODO

I'm doing the first smoke tests to see if the feature works as expected:
so far so good. Unit tests should be added before merging into the
4.0 branch.
2016-12-13 10:59:54 +01:00
Salvatore Sanfilippo f7c79ed474 Merge pull request #3680 from yossigo/fix_rediscli_command_crash
Fix redis-cli rare crash.
2016-12-12 19:36:15 +01:00
Yossi Gottlieb e4ec8f9b98 Fix redis-cli rare crash.
This happens if the server (mysteriously) returns an unexpected response
to the COMMAND command.
2016-12-12 20:18:40 +02:00
Jan-Erik Rediger 31624d82af Reset the ttl for additional keys
Before, if a previous key had a TTL set but the current one didn't, the
TTL was reused and thus resulted in wrong expirations set.

This behaviour was experienced, when `MigrateDefaultPipeline` in
redis-trib was set to >1

Fixes #3655
2016-12-08 14:27:21 +01:00
Salvatore Sanfilippo 53e5959568 Merge pull request #3663 from wshn13/add-LF-character-in-memory-doctor-output-message
Add '\n' to MEMORY DOCTOR command output message
2016-12-06 09:20:36 +01:00
wangshaonan a62b2a97a0 Add '\n' to MEMORY DOCTOR command output message when num_reports
is 0 or empty is 1
2016-12-06 03:11:27 +00:00
itamar c64b709956 Corrects a couple of omissions in the modules docs 2016-12-05 18:34:38 +02:00
antirez 4ca7de5ed8 Modules: types doc updated to new API. 2016-12-05 14:40:51 +01:00
antirez 9fd91769e9 Modules: API doc updated (auto generated). 2016-12-05 14:40:43 +01:00
antirez 80ce225981 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2016-12-05 14:17:11 +01:00
antirez 664cf834c1 Geo: improve fuzz test.
The test now uses more diverse radius sizes, especially sizes near or
greater the whole earth surface are used, that are known to trigger edge
cases. Moreover the PRNG seeding was probably resulting into the same
sequence tested over and over again, now seeding unsing the current unix
time in milliseconds.

Related to #3631.
2016-12-05 14:16:46 +01:00
antirez 81b343c386 Geo: fix computation of bounding box.
A bug was reported in the context in issue #3631. The root cause of the
bug was that certain neighbor boxes were zeroed after the "inside the
bounding box or not" check, simply because the bounding box computation
function was wrong.

A few debugging infos where enhanced and moved in other parts of the
code. A check to avoid steps=0 was added, but is unrelated to this
issue and I did not verified it was an actual bug in practice.
2016-12-05 14:02:32 +01:00
cbgbt 1a5a205969 cli: Only print elapsed time on OUTPUT_STANDARD 2016-12-02 20:59:33 -08:00
Itamar Haber d22d2b7b7f Verify pairs are provided after subcommands
Fixes https://github.com/antirez/redis/issues/3639
2016-12-02 18:19:36 +02:00
antirez acd9aea8bb PSYNC2: Do not accept WAIT in slave instances.
No longer makes sense since writable slaves only do local writes now:
writes are no longer passed to sub-slaves in the stream.
2016-12-02 10:21:20 +01:00
Chris Lamb 497fb6a0fd src/rdb.c: Correct "whenver" -> "whenever" typo. 2016-12-01 13:16:30 +01:00
Salvatore Sanfilippo 641209e3f3 Merge pull request #3651 from yossigo/datatype_methods_typo
Fix typo in RedisModuleTypeMethods declaration.
2016-12-01 09:09:37 +01:00
Yossi Gottlieb 729eef96f1 Fix typo in RedisModuleTypeMethods declaration. 2016-11-30 22:05:59 +02:00
Salvatore Sanfilippo dba456b73a Merge pull request #3648 from dvirsky/fix_reply_crash
fix memory corruption on RM_FreeCallReply
2016-11-30 11:21:10 +01:00
antirez a52b715835 Modules: change type registration API to use a struct of methods. 2016-11-30 11:14:01 +01:00
Dvir Volk 7a045a9a6b fix memory corruption on RM_FreeCallReply 2016-11-30 11:49:49 +02:00
antirez 02b570290e PSYNC2 test: check ability to resync after restart. 2016-11-29 11:15:16 +01:00
antirez 17c02cee7d PSYNC2 test: 20 seconds are enough... 2016-11-29 10:27:53 +01:00
antirez 855b0c05e7 PSYNC2 test: test added to the default tests. 2016-11-29 10:25:42 +01:00
antirez d262113e5d PSYNC2: Minor memory leak reading -NOMASTERLINK master reply fixed. 2016-11-29 10:25:00 +01:00
antirez 49af69dfdd PSYNC2 test: modify the test for production. 2016-11-29 10:22:40 +01:00
andyli f687d4fb98 Modify MIN->MAX 2016-11-29 16:34:41 +08:00
antirez 5639ca070b PSYNC2: stop sending newlines to sub-slaves when master is down.
This actually includes two changes:

1) No newlines to take the master-slave link up when the upstream master
is down. Doing this is dangerous because the sub-slave often is received
replication protocol for an half-command, so can't receive newlines
without desyncing the replication link, even with the code in order to
cancel out the bytes that PSYNC2 was using. Moreover this is probably
also not needed/sane, because anyway the slave can keep serving
requests, and because if it's configured to don't serve stale data, it's
a good idea, actually, to break the link.

2) When a +CONTINUE with a different ID is received, we now break
connection with the sub-slaves: they need to be notified as well. This
was part of the original specification but for some reason it was not
implemented in the code, and was alter found as a PSYNC2 bug in the
integration testing.
2016-11-28 17:54:04 +01:00
antirez 893f757ebc PSYNC2: Test (WIP).
This is the PSYNC2 test that helped find issues in the code, and that
still can show a protocol desync from time to time. Work is in progress
in order to find the issue. For now the test is not enabled in "make
test" and must be run manually.
2016-11-28 10:13:24 +01:00
antirez b429928687 Better protocol errors logging. 2016-11-25 10:55:16 +01:00
antirez a751287188 PSYNC2: on transient error jump to error, not write_error. 2016-11-24 15:48:18 +01:00
antirez 67a41fe316 Modules: fix client blocking calls access to invalid struct field.
We already have reference to the client pointer, no need to access the
already freed structure.

Close #3634.
2016-11-24 11:05:19 +01:00
antirez b2dc3a3d9c PSYNC2: bugfixing pre release.
1. Master replication offset was cleared after switching configuration
to some other slave, since it was assumed you can't PSYNC after a
switch. Note the case anymore and when we successfully PSYNC we need to
have our offset untouched.

2. Secondary replication ID was not reset to "000..." pattern at
startup.

3. Master in error state replying -LOADING or other transient errors
forced the slave to discard the cached master and full resync. This is
now fixed.

4. Better logging of what's happening on failed PSYNCs.
2016-11-23 17:36:45 +01:00
antirez 72c02940dd Merge branch 'unstable' of github.com:/antirez/redis into unstable 2016-11-18 13:10:57 +01:00
antirez eb5c80460e Test: WAIT tests added in wait.tcl unit. 2016-11-18 13:10:29 +01:00
Salvatore Sanfilippo 8b13f9ab86 Merge pull request #3612 from deep011/unstable
fix a possible bug for 'replconf getack'
2016-11-18 10:45:09 +01:00
antirez 04944544ca Merge branch 'psync2' into unstable 2016-11-17 09:37:03 +01:00
oranagra 1de180cd82 when a slave loads an RDB, stop an AOFRW fork before flusing db and parsing rdb file, to avoid a CoW disaster. 2016-11-16 21:30:59 +02:00
antirez a2d8c8e104 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2016-11-16 14:13:37 +01:00
antirez 704ab7234b Cluster: handle zero bytes at the end of nodes.conf. 2016-11-16 14:13:18 +01:00
deep011 c715f0e4be fix a possible bug for 'replconf getack' 2016-11-16 11:04:33 +08:00
hylepo 0f17efd238 Merge pull request #1 from hylepo/Fixing-typo-in-the-usage-of-redis-benchmark
Fixing typo in the usage of redis-benchmark
2016-11-11 10:34:37 +08:00
hylepo 43adf0a308 Update redis-benchmark.c
Fixing typo in the usage of redis-benchmark
2016-11-11 10:33:48 +08:00
oranagra 358e51d80a fix unsigned int overflow in adjustOpenFilesLimit 2016-11-10 16:59:52 +02:00
antirez ba7002b9bb PSYNC2: Save replication ID/offset on RDB file.
This means that stopping a slave and restarting it will still make it
able to PSYNC with the master. Moreover the master itself will retain
its ID/offset, in case it gets turned into a slave, or if a slave will
try to PSYNC with it with an exactly updated offset (otherwise there is
no backlog).

This change was possible thanks to PSYNC v2 that makes saving the current
replication state much simpler.
2016-11-10 12:35:29 +01:00
antirez cd5255eea4 PSYNC2: Wrap debugging code with if(0) 2016-11-09 15:37:15 +01:00
antirez ba91b5dde4 PSYNC2: different improvements to Redis replication.
The gist of the changes is that now, partial resynchronizations between
slaves and masters (without the need of a full resync with RDB transfer
and so forth), work in a number of cases when it was impossible
in the past. For instance:

1. When a slave is promoted to mastrer, the slaves of the old master can
partially resynchronize with the new master.

2. Chained slalves (slaves of slaves) can be moved to replicate to other
slaves or the master itsef, without requiring a full resync.

3. The master itself, after being turned into a slave, is able to
partially resynchronize with the new master, when it joins replication
again.

In order to obtain this, the following main changes were operated:

* Slaves also take a replication backlog, not just masters.

* Same stream replication for all the slaves and sub slaves. The
replication stream is identical from the top level master to its slaves
and is also the same from the slaves to their sub-slaves and so forth.
This means that if a slave is later promoted to master, it has the
same replication backlong, and can partially resynchronize with its
slaves (that were previously slaves of the old master).

* A given replication history is no longer identified by the `runid` of
a Redis node. There is instead a `replication ID` which changes every
time the instance has a new history no longer coherent with the past
one. So, for example, slaves publish the same replication history of
their master, however when they are turned into masters, they publish
a new replication ID, but still remember the old ID, so that they are
able to partially resynchronize with slaves of the old master (up to a
given offset).

* The replication protocol was slightly modified so that a new extended
+CONTINUE reply from the master is able to inform the slave of a
replication ID change.

* REPLCONF CAPA is used in order to notify masters that a slave is able
to understand the new +CONTINUE reply.

* The RDB file was extended with an auxiliary field that is able to
select a given DB after loading in the slave, so that the slave can
continue receiving the replication stream from the point it was
disconnected without requiring the master to insert "SELECT" statements.
This is useful in order to guarantee the "same stream" property, because
the slave must be able to accumulate an identical backlog.

* Slave pings to sub-slaves are now sent in a special form, when the
top-level master is disconnected, in order to don't interfer with the
replication stream. We just use out of band "\n" bytes as in other parts
of the Redis protocol.

An old design document is available here:

https://gist.github.com/antirez/ae068f95c0d084891305

However the implementation is not identical to the description because
during the work to implement it, different changes were needed in order
to make things working well.
2016-11-09 15:37:15 +01:00
Salvatore Sanfilippo 4ac7d7780d Merge pull request #3568 from MichaelTSS/patch-1
Typo
2016-11-02 15:18:44 +01:00
antirez 9050858254 redis-cli typo fixed: perferences -> preferences.
Thanks to @qiaodaimadelaowang for signaling the issue.
Close #3585.
2016-11-02 15:15:49 +01:00
Salvatore Sanfilippo e3645fafb4 Merge pull request #3514 from charsyam/feature/simple-refactoring
Simple change just using slaves instead of server.slaves
2016-11-02 11:04:52 +01:00
Salvatore Sanfilippo f4a291bdf2 Merge pull request #3547 from yyoshiki41/refactor/redis-trib
Refactor redis-trib.rb
2016-11-02 11:02:32 +01:00
Salvatore Sanfilippo d0f1ca15ec Merge pull request #3575 from deep011/unstable
fix a bug for quicklistDup() function
2016-11-02 11:00:24 +01:00
Salvatore Sanfilippo 19a5c67e92 Merge pull request #3581 from dvirsky/fix_io_ctx_sizeof
[MODULES] fixed sizeof in allocating io RedisModuleCtx*
2016-11-02 10:58:30 +01:00
Dvir Volk 3b7e7aa13b fixed sizeof in allocating io RedisModuleCtx* 2016-10-31 18:48:16 +02:00
antirez 47396311df Test: regression test for #3564 added. 2016-10-31 15:46:58 +01:00
Salvatore Sanfilippo 97c0733917 Merge pull request #3565 from sunheehnus/bitfield-fix-highest_write_offset
bitops.c/bitfieldCommand: update higest_write_offset with check
2016-10-31 15:40:46 +01:00
Salvatore Sanfilippo 84c9cbea61 Merge pull request #3573 from jybaek/module-io-context
Add missing fclose()
2016-10-31 15:36:38 +01:00
Salvatore Sanfilippo 958465f0b3 Merge pull request #3579 from guybe7/unstable
Fixed wrong sizeof(client) in object.c
2016-10-31 15:34:16 +01:00
Guy Benoish 7365268efa Fixed wrong sizeof(client) in object.c 2016-10-31 15:08:17 +02:00
deep 4504493445 fix a bug for quicklistDup() function 2016-10-28 19:47:29 +08:00
jybaek 2025738ce2 Add missing fclose() 2016-10-28 10:42:54 +09:00
Michel Tresseras c9ffc440c8 Typo 2016-10-24 17:27:48 +02:00
sunhe 0ab42c2c00 bitops.c/bitfieldCommand: update higest_write_offset with check 2016-10-22 01:54:46 +08:00
antirez f487fca502 Remove "Hey!" warning... 2016-10-19 10:43:40 +02:00
antirez 0a051ea1c1 Better target MacOS on __atomic macros conditional compilation. 2016-10-17 16:41:39 +02:00
Salvatore Sanfilippo ef45eeea0d Merge pull request #3560 from melo/fix-macos-10-8-compile
Fixes compilation on MacOS 10.8.5, Clang tags/Apple/clang-421.0.57
2016-10-17 16:11:00 +02:00
Pedro Melo 5d5ce20ee6 Fixes compilation on MacOS 10.8.5, Clang tags/Apple/clang-421.0.57
Redis fails to compile on MacOS 10.8.5 with Clang 4, version 421.0.57
(based on LLVM 3.1svn).

When compiling zmalloc.c, we get these warnings:

        CC zmalloc.o
    zmalloc.c:109:5: warning: implicit declaration of function '__atomic_add_fetch' is invalid in C99 [-Wimplicit-function-declaration]
        update_zmalloc_stat_alloc(zmalloc_size(ptr));
        ^
    zmalloc.c:75:9: note: expanded from macro 'update_zmalloc_stat_alloc'
            atomicIncr(used_memory,__n,used_memory_mutex); \
            ^
    ./atomicvar.h:57:37: note: expanded from macro 'atomicIncr'
    #define atomicIncr(var,count,mutex) __atomic_add_fetch(&var,(count),__ATOMIC_RELAXED)
                                        ^
    zmalloc.c:145:5: warning: implicit declaration of function '__atomic_sub_fetch' is invalid in C99 [-Wimplicit-function-declaration]
        update_zmalloc_stat_free(oldsize);
        ^
    zmalloc.c:85:9: note: expanded from macro 'update_zmalloc_stat_free'
            atomicDecr(used_memory,__n,used_memory_mutex); \
            ^
    ./atomicvar.h:58:37: note: expanded from macro 'atomicDecr'
    #define atomicDecr(var,count,mutex) __atomic_sub_fetch(&var,(count),__ATOMIC_RELAXED)
                                        ^
    zmalloc.c:205:9: warning: implicit declaration of function '__atomic_load_n' is invalid in C99 [-Wimplicit-function-declaration]
            atomicGet(used_memory,um,used_memory_mutex);
            ^
    ./atomicvar.h:60:14: note: expanded from macro 'atomicGet'
        dstvar = __atomic_load_n(&var,__ATOMIC_RELAXED); \
                 ^
    3 warnings generated.

Also on lazyfree.c:

        CC lazyfree.o
    lazyfree.c:68:13: warning: implicit declaration of function '__atomic_add_fetch' is invalid in C99 [-Wimplicit-function-declaration]
                atomicIncr(lazyfree_objects,1,lazyfree_objects_mutex);
                ^
    ./atomicvar.h:57:37: note: expanded from macro 'atomicIncr'
    #define atomicIncr(var,count,mutex) __atomic_add_fetch(&var,(count),__ATOMIC_RELAXED)
                                        ^
    lazyfree.c:111:5: warning: implicit declaration of function '__atomic_sub_fetch' is invalid in C99 [-Wimplicit-function-declaration]
        atomicDecr(lazyfree_objects,1,lazyfree_objects_mutex);
        ^
    ./atomicvar.h:58:37: note: expanded from macro 'atomicDecr'
    #define atomicDecr(var,count,mutex) __atomic_sub_fetch(&var,(count),__ATOMIC_RELAXED)
                                        ^
    2 warnings generated.

Then in the linking stage:

        LINK redis-server
    Undefined symbols for architecture x86_64:
      "___atomic_add_fetch", referenced from:
          _zmalloc in zmalloc.o
          _zcalloc in zmalloc.o
          _zrealloc in zmalloc.o
          _dbAsyncDelete in lazyfree.o
          _emptyDbAsync in lazyfree.o
          _slotToKeyFlushAsync in lazyfree.o
      "___atomic_load_n", referenced from:
          _zmalloc_used_memory in zmalloc.o
          _zmalloc_get_fragmentation_ratio in zmalloc.o
      "___atomic_sub_fetch", referenced from:
          _zrealloc in zmalloc.o
          _zfree in zmalloc.o
          _lazyfreeFreeObjectFromBioThread in lazyfree.o
          _lazyfreeFreeDatabaseFromBioThread in lazyfree.o
          _lazyfreeFreeSlotsMapFromBioThread in lazyfree.o
    ld: symbol(s) not found for architecture x86_64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    make[1]: *** [redis-server] Error 1
    make: *** [all] Error 2

With this patch, the compilation is sucessful, no warnings.

Running `make test` we get a almost clean bill of health. Test pass with
one exception:

    [err]: Check for memory leaks (pid 52793) in tests/unit/dump.tcl
    [err]: Check for memory leaks (pid 53103) in tests/unit/auth.tcl
    [err]: Check for memory leaks (pid 53117) in tests/unit/auth.tcl
    [err]: Check for memory leaks (pid 53131) in tests/unit/protocol.tcl
    [err]: Check for memory leaks (pid 53145) in tests/unit/protocol.tcl
    [ok]: Check for memory leaks (pid 53160)
    [err]: Check for memory leaks (pid 53175) in tests/unit/scan.tcl
    [ok]: Check for memory leaks (pid 53189)
    [err]: Check for memory leaks (pid 53221) in tests/unit/type/incr.tcl
    .
    .
    .

Full debug log (289MB, uncompressed) available at
https://dl.dropboxusercontent.com/u/75548/logs/redis-debug-log-macos-10.8.5.log.xz

Most if not all of the memory leak tests fail. Not sure if this is
related. They are the only ones that fail. I belive they are not related,
but just the memory leak detector is not working properly on 10.8.5.

Signed-off-by: Pedro Melo <melo@simplicidade.org>
2016-10-17 14:58:23 +01:00
antirez 96dad4da1f Fix SELECT test, broken cause change in error msg. 2016-10-14 15:48:11 +02:00
antirez a89f98b802 SWAPDB command.
This new command swaps two Redis databases, so that immediately all the
clients connected to a given DB will see the data of the other DB, and
the other way around. Example:

    SWAPDB 0 1

This will swap DB 0 with DB 1. All the clients connected with DB 0 will
immediately see the new data, exactly like all the clients connected
with DB 1 will see the data that was formerly of DB 0.

MOTIVATION AND HISTORY
---

The command was recently demanded by Pedro Melo, but was suggested in
the past multiple times, and always refused by me.

The reason why it was asked: Imagine you have clients operating in DB 0.
At the same time, you create a new version of the dataset in DB 1.
When the new version of the dataset is available, you immediately want
to swap the two views, so that the clients will transparently use the
new version of the data. At the same time you'll likely destroy the
DB 1 dataset (that contains the old data) and start to build a new
version, to repeat the process.

This is an interesting pattern, but the reason why I always opposed to
implement this, was that FLUSHDB was a blocking command in Redis before
Redis 4.0 improvements. Now we have FLUSHDB ASYNC that releases the
old data in O(1) from the point of view of the client, to reclaim memory
incrementally in a different thread.

At this point, the pattern can really be supported without latency
spikes, so I'm providing this implementation for the users to comment.
In case a very compelling argument will be made against this new command
it may be removed.

BEHAVIOR WITH BLOCKING OPERATIONS
---

If a client is blocking for a list in a given DB, after the swap it will
still be blocked in the same DB ID, since this is the most logical thing
to do: if I was blocked for a list push to list "foo", even after the
swap I want still a LPUSH to reach the key "foo" in the same DB in order
to unblock.

However an interesting thing happens when a client is, for instance,
blocked waiting for new elements in list "foo" of DB 0. Then the DB
0 and 1 are swapped with SWAPDB. However the DB 1 happened to have
a list called "foo" containing elements. When this happens, this
implementation can correctly unblock the client.

It is possible that there are subtle corner cases that are not covered
in the implementation, but since the command is self-contained from the
POV of the implementation and the Redis core, it cannot cause anything
bad if not used.

Tests and documentation are yet to be provided.
2016-10-14 15:28:04 +02:00
antirez 0d55b08656 Modules: use RedisModule_AbortBlock() in the example. 2016-10-13 17:00:45 +02:00
antirez a7af7a1f8e Modules: AbortBlock() API implemented. 2016-10-13 16:57:40 +02:00
antirez 6b22a3f9f2 Modules: blocking API documented. 2016-10-13 16:57:28 +02:00
antirez f34ceb42c0 module.c: trim comment to 80 cols. 2016-10-13 12:48:36 +02:00
antirez 626270fc82 Example modules: remove warnings about types and not used args. 2016-10-13 12:43:18 +02:00
jybaek 834a914b39 Remove Duplicate Processing 2016-10-13 15:17:07 +09:00
yyoshiki41 8e814916c3 Refactor redis-trib.rb 2016-10-10 01:13:20 +09:00
antirez 33223ded21 Modules: blocking command example added. 2016-10-07 16:35:06 +02:00
antirez 7ac5f45831 Modules: fixes to the blocking commands API: examples now works. 2016-10-07 16:34:40 +02:00
antirez 5a1a9e13f2 Modules: RM_Milliseconds() API added. 2016-10-07 16:34:19 +02:00
antirez e102e93c9d Modules: blocking commands WIP: API exported, a first example. 2016-10-07 13:48:14 +02:00
antirez 2371a8b3b3 Modules: introduce warning suppression macro for unused args. 2016-10-07 13:10:31 +02:00
antirez c10a839e82 Enable warning in example modules Makefile. 2016-10-07 13:07:13 +02:00
antirez 925752a13d Module: API to block clients with threading support.
Just a draft to align the main ideas, never executed code. Compiles.
2016-10-07 11:55:35 +02:00
antirez 9558e60f9c Fix typos in GetContextFromIO API declaration. 2016-10-06 18:26:04 +02:00
antirez 089e4bb667 Fix name of mispelled function. 2016-10-06 17:10:47 +02:00
antirez 6b0d47796f Module: Ability to get context from IO context.
It was noted by @dvirsky that it is not possible to use string functions
when writing the AOF file. This sometimes is critical since the command
rewriting may need to be built in the context of the AOF callback, and
without access to the context, and the limited types that the AOF
production functions will accept, this can be an issue.

Moreover there are other needs that we can't anticipate regarding the
ability to use Redis Modules APIs using the context in order to build
representations to emit AOF / RDB.

Because of this a new API was added that allows the user to get a
temporary context from the IO context. The context is auto released
if obtained when the RDB / AOF callback returns.

Calling multiple time the function to get the context, always returns
the same one, since it is invalid to have more than a single context.
2016-10-06 17:09:26 +02:00
antirez 185e565bec Copyright notice added to module.c. 2016-10-06 08:48:21 +02:00
antirez 80df924169 Modules: API to save/load single precision floating point numbers.
When double precision is not needed, to take 2x space in the
serialization is not good.
2016-10-03 00:08:35 +02:00
antirez 77e9a88be3 Modules: API to log from module I/O callbacks. 2016-10-02 16:51:37 +02:00
antirez 8c38deace1 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2016-10-02 16:50:37 +02:00
antirez f73d3e71dc Add compiler optimizations to example module makefile. 2016-10-02 11:01:36 +02:00
antirez 707413234f debug.c: include dlfcn.h regardless of BACKTRACE support. 2016-09-27 00:29:47 +02:00
antirez 89098019cf Merge branch 'unstable' of github.com:/antirez/redis into unstable 2016-09-26 09:10:52 +02:00
antirez 915683cee9 Security: CONFIG SET client-output-buffer-limit overflow fixed.
This commit fixes a vunlerability reported by Cory Duplantis
of Cisco Talos, see TALOS-2016-0206 for reference.

CONFIG SET client-output-buffer-limit accepts as client class "master"
which is actually only used to implement CLIENT KILL. The "master" class
has ID 3. What happens is that the global structure:

    server.client_obuf_limits[class]

Is accessed with class = 3. However it is a 3 elements array, so writing
the 4th element means to write up to 24 bytes of memory *after* the end
of the array, since the structure is defined as:

    typedef struct clientBufferLimitsConfig {
        unsigned long long hard_limit_bytes;
        unsigned long long soft_limit_bytes;
        time_t soft_limit_seconds;
    } clientBufferLimitsConfig;

EVALUATION OF IMPACT:

Checking what's past the boundaries of the array in the global
'server' structure, we find AOF state fields:

    clientBufferLimitsConfig client_obuf_limits[CLIENT_TYPE_OBUF_COUNT];
    /* AOF persistence */
    int aof_state;                  /* AOF_(ON|OFF|WAIT_REWRITE) */
    int aof_fsync;                  /* Kind of fsync() policy */
    char *aof_filename;             /* Name of the AOF file */
    int aof_no_fsync_on_rewrite;    /* Don't fsync if a rewrite is in prog. */
    int aof_rewrite_perc;           /* Rewrite AOF if % growth is > M and... */
    off_t aof_rewrite_min_size;     /* the AOF file is at least N bytes. */
    off_t aof_rewrite_base_size;    /* AOF size on latest startup or rewrite. */
    off_t aof_current_size;         /* AOF current size. */

Writing to most of these fields should be harmless and only cause problems in
Redis persistence that should not escalate to security problems.
However unfortunately writing to "aof_filename" could be potentially a
security issue depending on the access pattern.

Searching for "aof.filename" accesses in the source code returns many different
usages of the field, including using it as input for open(), logging to the
Redis log file or syslog, and calling the rename() syscall.

It looks possible that attacks could lead at least to informations
disclosure of the state and data inside Redis. However note that the
attacker must already have access to the server. But, worse than that,
it looks possible that being able to change the AOF filename can be used
to mount more powerful attacks: like overwriting random files with AOF
data (easily a potential security issue as demostrated here:
http://antirez.com/news/96), or even more subtle attacks where the
AOF filename is changed to a path were a malicious AOF file is loaded
in order to exploit other potential issues when the AOF parser is fed
with untrusted input (no known issue known currently).

The fix checks the places where the 'master' class is specifiedf in
order to access configuration data structures, and return an error in
this cases.

WHO IS AT RISK?

The "master" client class was introduced in Redis in Jul 28 2015.
Every Redis instance released past this date is not vulnerable
while all the releases after this date are. Notably:

    Redis 3.0.x is NOT vunlerable.
    Redis 3.2.x IS vulnerable.
    Redis unstable is vulnerable.

In order for the instance to be at risk, at least one of the following
conditions must be true:

    1. The attacker can access Redis remotely and is able to send
       the CONFIG SET command (often banned in managed Redis instances).

    2. The attacker is able to control the "redis.conf" file and
       can wait or trigger a server restart.

The problem was fixed 26th September 2016 in all the releases affected.
2016-09-26 08:47:52 +02:00
charsyam 00a4156fdf Simple change just using slaves instead of server.slaves 2016-09-24 15:53:57 +09:00
Salvatore Sanfilippo 22e1b0a6a4 Merge pull request #3511 from dvirsky/create_string_printf
added RM_CreateStringPrintf
2016-09-21 11:46:53 +02:00
Dvir Volk 57009a1ebe added RM_CreateStringPrintf 2016-09-21 12:30:38 +03:00
antirez cc88e14c87 dict.c: fix dictGenericDelete() return ASAP condition.
Recently we moved the "return ASAP" condition for the Delete() function
from checking .size to checking .used, which is smarter, however while
testing the first table alone always works to ensure the dict is totally
emtpy, when we test the .size field, testing .used requires testing both
T0 and T1, since a rehashing could be in progress.
2016-09-20 17:22:30 +02:00
antirez de5aa838dd Clear child data when opening the pipes.
This is important both to reset the magic to 0, so that it will not
match if the structure is not explicitly set, and to initialize other
things we may add like counters and such.
2016-09-19 14:11:17 +02:00
antirez 5ac4559da3 Child -> Parent pipe for COW info transferring. 2016-09-19 13:45:20 +02:00
antirez 477d5e0c70 zmalloc: Make fp var non local to fix build. 2016-09-19 10:34:39 +02:00
antirez 1525b7380f zmalloc: zmalloc_get_smap_bytes_by_field() modified to work for any PID.
The goal is to get copy-on-write amount of the child from the parent.
2016-09-19 10:28:42 +02:00
antirez f8eb871244 redis-cli: "allocator-stats" -> "malloc-stats".
It was changed in Redis but not in redis-cli.
Thanks to @oranagra for signaling.
2016-09-19 09:47:35 +02:00
antirez 0bbc678616 Typo fixed from MEMORY DOCTOR output. 2016-09-16 16:52:00 +02:00
antirez 18a863b911 Surround allocator name with quotes in MEMORY DOCTOR output. 2016-09-16 16:40:25 +02:00
antirez e31078c7b1 MEMORY DOCTOR initial implementation. 2016-09-16 16:36:53 +02:00
antirez 29a75e3d62 Provide percentage of memory peak used info. 2016-09-16 10:43:19 +02:00
oranagra 00c0c40ba5 add zmalloc used mem to DEBUG SDSLEN 2016-09-16 10:29:27 +02:00
antirez 03c1271cf2 Memory related subcommands of DEBUG moved to MEMORY. 2016-09-16 10:26:23 +02:00
antirez d28b77b15f Group MEMORY command related APIs together in the source code. 2016-09-16 10:12:04 +02:00
antirez 65281a94a5 objectComputeSize(): skiplist nodes have different sizes.
The size of the node depends on the node level, however it is not stored
into the node itself, is an implicit information, so we use
zmalloc_size() in order to compute the sorted set size.
2016-09-15 17:43:13 +02:00
antirez a08f8acfc4 MEMORY command: HELP + dataset percentage (like in INFO). 2016-09-15 17:33:16 +02:00
antirez 033a49e23d MEMORY USAGE: SAMPLES option added + fixes to size computation.
The new SAMPLES option is added, defaulting to 5, and with 0 being a
special value to scan the whole set of elements.

Fixes to the object size computation were made since the original PR
assumed data structures still contaning robj structures, while now after
the lazyfree changes, are all SDS strings.
2016-09-15 15:25:14 +02:00
antirez a427affe30 INFO: new memory reporting fields added. 2016-09-15 10:33:23 +02:00
antirez 2eeedfe97f C struct memoh renamed redisMemOverhead. API prototypes added. 2016-09-15 09:44:07 +02:00
antirez 8a81227859 MEMORY OVERHEAD refactored into a generic API. 2016-09-15 09:37:55 +02:00
antirez 42acf62e5f dict.c: dictReplaceRaw() -> dictAddOrFind().
What they say about "naming things" in programming?
2016-09-14 16:43:38 +02:00
antirez 28942ddac7 Trim comment to 80 cols. 2016-09-14 16:41:05 +02:00
antirez e027e9e85d Apply the new dictUnlink() where possible.
Optimizations suggested and originally implemented by @oranagra.
Re-applied by @antirez using the modified API.
2016-09-14 16:37:53 +02:00
oranagra 40cf4d9a0a dict.c: introduce dictUnlink().
Notes by @antirez:

This patch was picked from a larger commit by Oran and adapted to change
the API a bit. The basic idea is to avoid double lookups when there is
to use the value of the deleted entry.

BEFORE:

    entry = dictFind( ... ); /* 1st lookup. */
    /* Do somethjing with the entry. */
    dictDelete(...);         /* 2nd lookup. */

AFTER:

    entry = dictUnlink( ... ); /* 1st lookup. */
    /* Do somethjing with the entry. */
    dictFreeUnlinkedEntry(entry); /* No lookups!. */
2016-09-14 12:18:59 +02:00
antirez a114054182 MEMORY OVERHEAD implemented (using Oran Agra initial implementation).
This code was extracted from @oranagra PR #3223 and modified in order
to provide only certain amounts of information compared to the original
code. It was also moved from DEBUG to the newly introduced MEMORY
command. Thanks to Oran for the implementation and the PR.

It implements detailed memory usage stats that can be useful in both
provisioning and troubleshooting memory usage in Redis.
2016-09-13 17:39:25 +02:00
antirez 33ed0a71cd objectComputeSize(): estimate collections sampling N elements.
For most tasks, we need the memory estimation to be O(1) by default.
This commit also implements an initial MEMORY command.
Note that objectComputeSize() takes the number of samples to check as
argument, so MEMORY should be able to get the sample size as option
to make precision VS CPU tradeoff tunable.

Related to: PR #3223.
2016-09-13 10:28:23 +02:00
oranagra d028418790 Adding objectComputeSize() function. 2016-09-12 16:36:59 +02:00
oranagra 1ef16debfb Optimize repeated keyname hashing.
(Change cherry-picked and modified by @antirez from a larger commit
provided by @oranagra in PR #3223).
2016-09-12 13:19:05 +02:00
Alexander Zhukov 1f3cd7087f Fix an article usage 2016-09-12 13:53:47 +03:00
Salvatore Sanfilippo 53530e09c5 Merge pull request #3492 from wyxustcsa09/fix-memory
fix memory error on module unload
2016-09-09 16:05:06 +02:00
antirez 6a47311dfc Merge branch 'unstable' of github.com:antirez/redis into unstable 2016-09-09 16:01:43 +02:00
antirez c72557349a Example modules: Add C99 standard to cflags. 2016-09-09 16:01:29 +02:00
antirez 125deba083 Merge branch 'aofrdb' into unstable 2016-09-09 15:03:21 +02:00
antirez 9edd074a1d fix the fix for the TCP binding.
This commit attempts to fix a problem with PR #3467.
2016-09-09 14:59:48 +02:00
oranagra c7dbd7d6c5 fix tcp binding when IPv6 is unsupported 2016-09-09 14:59:21 +02:00
antirez fef4253c31 debug.c: no need to define _GNU_SOURCE, is defined in fmacros.h. 2016-09-09 11:15:10 +02:00
antirez de5ac38179 crash log - improve code dump with more info and called symbols. 2016-09-09 11:00:19 +02:00
wyx 3e4e2fd08a fix memory error on module unload 2016-09-09 10:22:57 +08:00
oranagra 7698b97ca5 crash log - add hex dump of function code 2016-09-08 14:14:57 +02:00
antirez 229466fdf1 dict.c benchmark minor improvements. 2016-09-07 15:28:40 +02:00
antirez 259a909168 dict.c benchmark: mixed del/insert benchmark. 2016-09-07 12:34:53 +02:00
antirez a65fd55d3f dict.c benchmark: finish rehashing before testing lookups. 2016-09-07 11:06:03 +02:00
antirez b97146827d dict.c benchmark improvements. 2016-09-07 10:53:47 +02:00
antirez bd8a490194 dict.c benchmark: take optional count argument. 2016-09-07 10:44:29 +02:00
antirez ab03414ea5 dict.c benchmark. 2016-09-07 10:33:15 +02:00
antirez b462b22e59 Fix rdb.c var types when calling rdbLoadLen().
Technically as soon as Redis 64 bit gets proper support for loading
collections and/or DBs with more than 2^32 elements, the 32 bit version
should be modified in order to check if what we read from rdbLoadLen()
overflows. This would only apply to huge RDB files created with a 64 bit
instance and later loaded into a 32 bit instance.
2016-09-01 11:08:44 +02:00
antirez 982a34d067 sds: don't check for impossible string size in 32 bit systems. 2016-09-01 11:04:22 +02:00
antirez cf5154f72e RDB AOF preamble: test it in the aofrw unit. 2016-08-24 15:39:39 +02:00
antirez cfe0810936 Document RDB preamble in AOF rewrites in redis.conf. 2016-08-24 15:33:44 +02:00
antirez 02a360eca0 Sentinel example config: warn about protected mode. 2016-08-11 19:53:12 +02:00
antirez 982df150f8 RDB AOF preamble: WIP 4 (Mixed RDB/AOF loading). 2016-08-11 15:42:28 +02:00
antirez 96d57f8d55 RDB AOF preamble: WIP 3 (RDB loading refactoring). 2016-08-11 15:27:29 +02:00
antirez 95ce1aa378 RDB AOF preamble: WIP 2. 2016-08-09 16:41:40 +02:00
antirez 3bf40950e1 RDB AOF preamble: WIP 1. 2016-08-09 11:07:32 +02:00
Salvatore Sanfilippo 9200d87d30 Merge pull request #3340 from rojingeorge/unstable
Display the nodes' proper summary once the cluster is created using redis-trib
2016-08-05 15:10:41 +02:00
Salvatore Sanfilippo 27d1f4ed7c Merge pull request #3429 from guoxiao/warning
Use the standard predefined identifier __func__ (since C99)
2016-08-04 10:25:53 +02:00
Guo Xiao eb3640db00 Use the standard predefined identifier __func__ (since C99)
Fix warning: ISO C does not support '__FUNCTION__' predefined identifier
[-Wpedantic]
2016-08-04 15:12:12 +08:00
antirez 3ed474aa05 Modules: basic call/reply tests in test module. 2016-08-03 18:10:11 +02:00
antirez 935fb1eae2 Modules: handle NULL replies more gracefully.
After all crashing at every API misuse makes everybody's life more
complex.
2016-08-03 18:09:36 +02:00
antirez 954ac5c756 Security: Cross Protocol Scripting protection.
This is an attempt at mitigating problems due to cross protocol
scripting, an attack targeting services using line oriented protocols
like Redis that can accept HTTP requests as valid protocol, by
discarding the invalid parts and accepting the payloads sent, for
example, via a POST request.

For this to be effective, when we detect POST and Host: and terminate
the connection asynchronously, the networking code was modified in order
to never process further input. It was later verified that in a
pipelined request containing a POST command, the successive commands are
not executed.
2016-08-03 11:12:32 +02:00
antirez fa5d8e218e Fix comment over 80 cols. 2016-08-03 10:56:26 +02:00
antirez 89e24b8f4a Modules: initial draft for a testing module. 2016-08-03 10:23:03 +02:00
antirez 5a52229249 Modules: StringAppendBuffer() and ability to retain strings.
RedisModule_StringRetain() allows, when automatic memory management is
on, to keep string objects living after the callback returns. Can also
be used in order to use Redis reference counting of objects inside
modules.

The reason why this is useful is that sometimes when implementing new
data types we want to reference RedisModuleString objects inside the
module private data structures, so those string objects must be valid
after the callback returns even if not referenced inside the Redis key
space.
2016-08-02 15:29:04 +02:00
Qu Chen 7d67db4b9f Fix a bug to delay bgsave while AOF rewrite in progress for replication 2016-08-02 10:44:33 +02:00
antirez 964cea9fba Remove extra "-" from ASCII horizontal bar in comment. 2016-08-02 10:32:44 +02:00
antirez 53cc8d7ce6 Update linenoise to fix insecure redis-cli history file creation.
The problem was fixed in antirez/linenoise repository applying a patch
contributed by @lamby. Here the new version is updated in the Redis
source tree.

Close #1418
Close #3322
2016-07-29 11:28:16 +02:00
antirez 2c80701c7d Changelog format modified to be less verbose. 2016-07-28 14:15:31 +02:00
antirez f8a3861465 Ability of slave to announce arbitrary ip/port to master.
This feature is useful, especially in deployments using Sentinel in
order to setup Redis HA, where the slave is executed with NAT or port
forwarding, so that the auto-detected port/ip addresses, as listed in
the "INFO replication" output of the master, or as provided by the
"ROLE" command, don't match the real addresses at which the slave is
reachable for connections.
2016-07-27 17:32:15 +02:00
antirez 976c2425c4 Multiple GEORADIUS bugs fixed.
By grepping the continuous integration errors log a number of GEORADIUS
tests failures were detected.

Fortunately when a GEORADIUS failure happens, the test suite logs enough
information in order to reproduce the problem: the PRNG seed,
coordinates and radius of the query.

By reproducing the issues, three different bugs were discovered and
fixed in this commit. This commit also improves the already good
reporting of the fuzzer and adds the failure vectors as regression
tests.

The issues found:

1. We need larger squares around the poles in order to cover the area
requested by the user. There were already checks in order to use a
smaller step (larger squares) but the limit set (+/- 67 degrees) is not
enough in certain edge cases, so 66 is used now.

2. Even near the equator, when the search area center is very near the
edge of the square, the north, south, west or ovest square may not be
able to fully cover the specified radius. Now a test is performed at the
edge of the initial guessed search area, and larger squares are used in
case the test fails.

3. Because of rounding errors between Redis and Tcl, sometimes the test
signaled false positives. This is now addressed.

Whenever possible the original code was improved a bit in other ways. A
debugging example stanza was added in order to make the next debugging
session simpler when the next bug is found.
2016-07-27 11:34:25 +02:00
antirez 0de6a2702b Replication: when possible start RDB saving ASAP.
In a previous commit the replication code was changed in order to
centralize the BGSAVE for replication trigger in replicationCron(),
however after further testings, the 1 second delay imposed by this
change is not acceptable.

So now the BGSAVE is only delayed if the AOF rewriting process is
active. However past comments made sure that replicationCron() is always
able to trigger the BGSAVE when needed, making the code generally more
robust.

The new code is more similar to the initial @oranagra patch where the
BGSAVE was delayed only if an AOF rewrite was in progress.

Trivia: delaying the BGSAVE uncovered a minor Sentinel issue that is now
fixed.
2016-07-22 17:03:18 +02:00
antirez 7534ce9996 Sentinel: new test unit 07 that tests master down conditions. 2016-07-22 16:39:26 +02:00
antirez f844032233 Sentinel: check Slave INFO state more often when disconnected.
During the initial handshake with the master a slave will report to have
a very high disconnection time from its master (since technically it was
disconnected since forever, so the current UNIX time in seconds is
reported).

However when the slave is connected again the Sentinel may re-scan the
INFO output again only after 10 seconds, which is a long time. During
this time Sentinels will consider this instance unable to failover, so
a useless delay is introduced.

Actaully this hardly happened in the practice because when a slave's
master is down, the INFO period for slaves changes to 1 second. However
when a manual failover is attempted immediately after adding slaves
(like in the case of the Sentinel unit test), this problem may happen.

This commit changes the INFO period to 1 second even in the case the
slave's master is not down, but the slave reported to be disconnected
from the master (by publishing, last time we checked, a master
disconnection time field in INFO).

This change is required as a result of an unrelated change in the
replication code that adds a small delay in the master-slave first
synchronization.
2016-07-22 10:51:25 +02:00
antirez 8f6844d7dd Avoid simultaneous RDB and AOF child process.
This patch, written in collaboration with Oran Agra (@oranagra) is a companion
to 780a8b1. Together the two patches should avoid that the AOF and RDB saving
processes can be spawned at the same time. Previously conditions that
could lead to two saving processes at the same time were:

1. When AOF is enabled via CONFIG SET and an RDB saving process is
   already active.

2. When the SYNC command decides to start an RDB saving process ASAP in
   order to serve a new slave that cannot partially resynchronize (but
   only if we have a disk target for replication, for diskless
   replication there is not such a problem).

Condition "1" is not very severe but "2" can happen often and is
definitely good at degrading Redis performances in an unexpected way.

The two commits have the effect of always spawning RDB savings for
replication in replicationCron() instead of attempting to start an RDB
save synchronously. Moreover when a BGSAVE or AOF rewrite must be
performed, they are instead just postponed using flags that will try to
perform such operations ASAP.

Finally the BGSAVE command was modified in order to accept a SCHEDULE
option so that if an AOF rewrite is in progress, when this option is
given, the command no longer returns an error, but instead schedules an
RDB rewrite operation for when it will be possible to start it.
2016-07-21 18:35:01 +02:00
antirez 1c2fb8a0df Replication: start BGSAVE for replication always in replicationCron().
This makes the replication code conceptually simpler by removing the
synchronous BGSAVE trigger in syncCommand(). This also means that
socket and disk BGSAVE targets are handled by the same code.
2016-07-21 12:10:56 +02:00
antirez b98c2df5a0 Fix maxmemory shared integer check bug introduced with LFU. 2016-07-21 11:14:18 +02:00
antirez 4376529929 Volatile-ttl eviction policy implemented in terms of the pool.
Precision of the eviction improved sensibly. Also this allows us to have
a single code path for most eviction types.
2016-07-20 19:54:12 +02:00
antirez 760725aa5b test-lru.rb: support for testing volatile-ttl policy. 2016-07-20 19:02:20 +02:00
antirez 0e1b0d6276 LFU: make counter log factor and decay time configurable. 2016-07-20 15:00:35 +02:00
antirez ab78df558e LFU: Use the LRU pool for the LFU algorithm.
Verified to have better real world performances with power-law access
patterns because of the data accumulated across calls.
2016-07-18 18:17:59 +02:00
antirez c262acb656 LFU: Fix bugs in frequency decay code. 2016-07-18 14:19:38 +02:00
antirez 4bc127a2bd LFU: Initial naive eviction cycle.
It is possible to get better results by using the pool like in the LRU
case. Also from tests during the morning I believe the current
implementation has issues in the frequency decay function that should
decrease the counter at periodic intervals.
2016-07-18 13:50:19 +02:00
antirez f934b0bbae redis-cli LRU test mode: randomize value of key when setting.
This way it is possible from an observer to tell when the key is
replaced with a new one having the same name.
2016-07-18 10:56:47 +02:00
antirez a7cd95aa8b redis-cli LRU test mode: remove newline from key names. 2016-07-18 10:53:02 +02:00
antirez 13c8765e00 LFU: Redis object level implementation.
Implementation of LFU maxmemory policy for anything related to Redis
objects. Still no actual eviction implemented.
2016-07-15 12:12:58 +02:00
antirez 8192fc3a8c LFU simulator: remove dead code. 2016-07-14 16:06:36 +02:00
antirez 1aaeeaa537 LRU simulator: fix new entry creation decr time. 2016-07-14 15:55:17 +02:00
antirez c8a6d75f1c LRU simulator: fix new entry creation. 2016-07-14 15:51:51 +02:00
antirez cd969573ed LFU: Simulation of the algorithm planned for Redis.
We have 24 total bits of space in each object in order to implement
an LFU (Least Frequently Used) eviction policy.

We split the 24 bits into two fields:

      8 bits      16 bits
    +--------+----------------+
    | LOG_C  | Last decr time |
    +--------+----------------+

LOG_C is a logarithmic counter that provides an indication of the access
frequency. However this field must also be deceremented otherwise what used
to be a frequently accessed key in the past, will remain ranked like that
forever, while we want the algorithm to adapt to access pattern changes.

So the remaining 16 bits are used in order to store the "decrement time",
a reduced-precision unix time (we take 16 bits of the time converted
in minutes since we don't care about wrapping around) where the LOG_C
counter is halved if it has an high value, or just decremented if it
has a low value.

New keys don't start at zero, in order to have the ability to collect
some accesses before being trashed away, so they start at COUNTER_INIT_VAL.
The logaritmic increment performed on LOG_C takes care of COUNTER_INIT_VAL
when incrementing the key, so that keys starting at COUNTER_INIT_VAL
(or having a smaller value) have a very high chance of being incremented
on access.

The simulation starts with a power-law access pattern, and later converts
into a flat access pattern in order to see how the algorithm adapts.
Currenty the decrement operation period is 1 minute, however note that
it is not guaranteed that each key will be scanned 1 time every minute,
so the actual frequency can be lower. However under high load, we access
3/5 keys every newly inserted key (because of how Redis eviction works).

This is a work in progress at this point to evaluate if this works well.
2016-07-14 15:21:48 +02:00
antirez bb5b1c5991 LRU: Make cross-database choices for eviction.
The LRU eviction code used to make local choices: for each DB visited it
selected the best key to evict. This was repeated for each DB. However
this means that there could be DBs with very frequently accessed keys
that are targeted by the LRU algorithm while there were other DBs with
many better candidates to expire.

This commit attempts to fix this problem for the LRU policy. However the
TTL policy is still not fixed by this commit. The TTL policy will be
fixed in a successive commit.

This is an initial (partial because of TTL policy) fix for issue #2647.
2016-07-13 13:12:30 +02:00
antirez 8934a2a78a LRU: cache SDS strings in the eviction pool.
To destroy and recreate the pool[].key element is slow, so we allocate
in pool[].cached SDS strings that can account up to 255 chars keys and
try to reuse them. This provides a solid 20% performance improvement
in real world workload alike benchmarks.
2016-07-12 12:31:37 +02:00
antirez 8464262b28 Move the struct evictionPoolEntry() into only file using it.
Local scope is always better when possible.
2016-07-12 12:22:38 +02:00
antirez 4ef07b3f9d Move prototype of evictionPoolAlloc() in server.h. 2016-07-12 12:22:35 +02:00
antirez 9f47c5a53b LRU: use C99 variable len stack array in evictionPoolPopulate(). 2016-07-12 12:05:45 +02:00
antirez f07cc50f89 redis-benchmark: new option to show server errors on stdout.
Disabled by default, can be activated with -e. Maybe the reverse was
more safe but departs from the past behavior.
2016-07-12 11:23:31 +02:00
antirez f708bf2a14 Remove useless memmove() from freeMemoryIfNeeded().
We start from the end of the pool to the initial item, zero-ing
every entry we use or every ghost entry, there is nothing to memmove
since to the right everything should be already set to NULL.
2016-07-11 19:18:17 +02:00
antirez bbac261b64 LRU: Fix output fixes to new test-lru.rb. 2016-07-11 16:26:02 +02:00
antirez 22be3d06e3 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2016-07-11 16:24:21 +02:00
antirez c35d315148 LRU: test-lru.rb improved in different ways.
1. Scan keys with pause to account for actual LRU precision.
2. Test cross-DB with 100 keys allocated in DB1.
3. Output results that don't fluctuate depending on number of keys.
4. Output results in percentage to make more sense.
5. Save file instead of outputting to STDOUT.
6. Support running multiple times with average of outputs.
7. Label each square (DIV) with its ID as HTML title.
2016-07-11 16:23:50 +02:00
antirez 2e7ab9a360 redis_check_rdb(): the rio structure must be global.
The rio structure is referenced in the global 'riostate' structure
in order for the logging functions to be always able to access the state
of the "pseudo-loading" of the RDB, needed for the check.

Courtesy of Valgrind.
2016-07-06 19:12:24 +02:00
antirez 3b5a3c7249 redis_check_rdb_main(): create shared objects only if needed.
Otherwise Valgrind will complain a memory leak under certain tests where
RDB checking is invoked from within Redis.
2016-07-06 19:06:22 +02:00
antirez 4e3fea4f1a Fix redis_check_rdb() return value. 2016-07-06 19:00:43 +02:00
antirez 6430ab2861 Remove dead code from geohash_helper.c.
The function removed also had potential bugs related to signess of the
expression, and is not used anyway.
2016-07-06 16:39:23 +02:00
antirez ceaa949cb8 Fix signess issue in geohashEstimateStepsByRadius(). 2016-07-06 16:38:05 +02:00
antirez 173cdcaccd Fix definition of M_PI in geohash_helper.c.
Without the right feature macros M_PI is not defined in math.h.
2016-07-06 16:31:11 +02:00
antirez d7322b6f2f geohash.c and geohash_helper.c are part of Redis.
They were under /deps since they originate from a different source tree,
however at this point they are very modified and we took ownership of
both the files making changes, fixing bugs, so there is no upgrade path
from the original code tree.

Given that, better to move the code under /src with proper dependencies
and with a more simpler editing experience.
2016-07-06 16:02:38 +02:00
antirez 0ed805f7f4 Add expire.c and evict.c. 2016-07-06 15:28:18 +02:00
antirez 0df4deada4 Expire and LRU related code moved into different files. 2016-07-06 15:24:06 +02:00
antirez d14ec7e975 Makefile: don't build dependencies file for clean, distclean. 2016-07-06 12:56:43 +02:00
antirez 262bc9d095 Generate Makefile.dep at every build.
Normally we used to update it from time to time. Too fragile... better
to generate dependencies at every run and delete them on 'make clean'.
2016-07-06 12:24:48 +02:00
antirez 669db62ae1 Regression test for issue #3333. 2016-07-06 11:50:20 +02:00
antirez d1f5aa7f87 getLongLongFromObject: use string2ll() instead of strict_strtoll().
strict_strtoll() has a bug that reports the empty string as ok and
parses it as zero.

Apparently nobody ever replaced this old call with the faster/saner
string2ll() which is used otherwise in the rest of the Redis core.

This commit close #3333.
2016-07-06 11:43:33 +02:00
antirez 14ee3418e5 redis-cli: check SELECT reply type just in state updated.
In issues #3361 / #3365 a problem was reported / fixed with redis-cli
not updating correctly the current DB on error after SELECT.

In theory this bug was fixed in 0042fb0e, but actually the commit only
fixed the prompt updating, not the fact the state was set in a wrong
way.

This commit removes the check in the prompt update, now that hopefully
it is the state that is correct, there is no longer need for this check.
2016-07-05 15:18:40 +02:00
Salvatore Sanfilippo 151044871c Merge pull request #3365 from sskorgal/unstable
Fix for redis_cli printing default DB when select command fails. #3361
2016-07-05 15:12:06 +02:00
antirez b5cef76b74 Sentinel: fix cross-master Sentinel address update.
This commit both fixes the crash reported with issue #3364 and
also properly closes the old links after the Sentinel address for the
other masters gets updated.

The two problems where:

1. The Sentinel that switched address may not monitor all the masters,
   it is possible that there is no match, and the 'match' variable is
   NULL. Now we check for no match and 'continue' to the next master.

2. By ispecting the code because of issue "1" I noticed that there was a
   problem in the code that disconnects the link of the Sentinel that
   needs the address update. Basically link->disconnected is non-zero
   even if just *a single link* (cc -- command link or pc -- pubsub
   link) are disconnected, so to check with if (link->disconnected)
   in order to close the links risks to leave one link connected.

I was able to manually reproduce the crash at "1" and verify that the
commit resolves the issue.

Close #3364.
2016-07-04 18:45:24 +02:00
antirez e1ecc3ab8e CONFIG GET is now no longer case sensitive.
Like CONFIG SET always was. Close #3369.
2016-07-04 16:09:24 +02:00
antirez 1ce90bd39f Fix test for new RDB checksum failure message. 2016-07-04 12:41:35 +02:00
antirez f503ca0c69 Make tcp-keepalive default to 300 in internal conf.
We already changed the default in the redis.conf template, but I forgot
to change the internal config as well.
2016-07-04 12:08:42 +02:00
antirez 3ede1e6d95 In Redis RDB check: more details in error reportings. 2016-07-01 15:26:55 +02:00
antirez ae3ce0eba3 In Redis RDB check: log decompression errors. 2016-07-01 11:59:25 +02:00
antirez 247cf937bb In Redis RDB check: log object type on error. 2016-07-01 11:40:40 +02:00
antirez d882ccb6a2 Added a trivial program to randomly corrupt RDB files in /utils. 2016-07-01 09:55:50 +02:00
antirez 80920cba73 In Redis RDB check: minor output message changes. 2016-07-01 09:52:35 +02:00
antirez c67f1a46e3 In Redis RDB check: better error reporting. 2016-07-01 09:36:52 +02:00
sskorgal 9aacdebf63 Fix for redis_cli printing default DB when select command fails. 2016-07-01 10:42:22 +05:30
antirez 3d4399e160 In Redis RDB check: initial POC.
So far we used an external program (later executed within Redis) and
parser in order to check RDB files for correctness. This forces, at each
RDB format update, to have two copies of the same format implementation
that are hard to keep in sync. Morover the former RDB checker only
checked the very high-level format of the file, without actually trying
to load things in memory. Certain corruptions can only be handled by
really loading key-value pairs.

This first commit attempts to unify the Redis RDB loadig code with the
task of checking the RDB file for correctness. More work is needed but
it looks like a sounding direction so far.
2016-06-30 23:44:44 +02:00
Rojin George 5b26a16096 Merge remote-tracking branch 'refs/remotes/antirez/unstable' into unstable 2016-06-30 16:34:01 +05:30
Yusaku Kaneta 679290d632 Fix the firstkey, lastkey, and keystep of moduleCommand 2016-06-29 18:09:16 +09:00
antirez 163f95978b Test: new randomized stress tester for #3343 alike bugs. 2016-06-28 09:42:20 +02:00
antirez 30e2f37ab7 Stress tester WIP. 2016-06-28 09:33:36 +02:00
antirez 2f3b3ae896 Regression test for issue #3343 exact min crash sequence.
Note: it was verified that it can crash the test suite without the patch
applied.
2016-06-28 09:27:14 +02:00
antirez 52e759540e Merge branch 'unstable' of github.com:/antirez/redis into unstable 2016-06-27 18:12:46 +02:00
antirez c147eaf8f8 Fix quicklistReplaceAtIndex() by updating the quicklist ziplist size.
The quicklist takes a cached version of the ziplist representation size
in bytes. The implementation must update this length every time the
underlying ziplist changes. However quicklistReplaceAtIndex() failed to
fix the length.

During LSET calls, the size of the ziplist blob and the cached size
inside the quicklist diverged. Later, when this size is used in an
authoritative way, for example during nodes splitting in order to copy
the nodes, we end with a duplicated node that may contain random
garbage.

This commit should fix issue #3343, however several problems were found
reviewing the quicklist.c code in search of this bug that should be
addressed soon or later.

For example:

1. To take a cached ziplist length is fragile since failing to update it
leads to this kind of issues.

2. The node splitting code needs auditing. For example it works just for
a side effect of ziplistDeleteRange() to be able to cope with a wrong
count of elements to remove. The code inside quicklist.c assumes that
-1 means "delete till the end" while actually it's just a count of how
many elements to delete, and is an unsigned count. So -1 gets converted
into the maximum integer, and just by chance the ziplist code stops
deleting elements after there are no more to delete.

3. Node splitting is extremely inefficient, it copies the node and
removes elements from both nodes even when actually there is to move a
single entry from one node to the other, or when the new resulting node
is empty at all so there is nothing to copy but just to create a new
node.

However at least for Redis 3.2 to introduce fresh code inside
quicklist.c may be even more risky, so instead I'm writing a better
fuzzy tester to stress the internals a bit more in order to anticipate
other possible bugs.

This bug was found using a fuzzy tester written after having some clue
about where the bug could be. The tester eventually created a ~2000
commands sequence able to always crash Redis. I wrote a better version
of the tester that searched for the smallest sequence that could crash
Redis automatically. Later this smaller sequence was minimized by
removing random commands till it still crashed the server. This resulted
into a sequence of 7 commands. With this small sequence it was just a
matter of filling the code with enough printf() to understand enough
state to fix the bug.
2016-06-27 18:02:33 +02:00
Salvatore Sanfilippo c40bd5fd61 Merge pull request #3342 from yossigo/fix_calloc
Fix RedisModule_Calloc() definition typo.
2016-06-24 08:58:23 +02:00
Yossi Gottlieb 0c94a88075 Fix RedisModule_Calloc() definition typo. 2016-06-23 22:30:32 +03:00
rojingeorge feee7d547a Display the nodes summary once the cluster is established using redis-trib.rb
Display the nodes summary once the cluster is established using
redis-trib.rb
After the cluster meet and join was done, when the summary was shown, it
was giving info regarding the nodes. This fix ensures that confusion
where the slaves were shown as masters.
Fix would be to reset the nodes and reload the cluster information
before checking the cluster status after creating it.
2016-06-23 21:58:03 +05:30
rojingeorge fae1ce2316 Merge remote-tracking branch 'refs/remotes/antirez/unstable' into unstable 2016-06-23 21:34:03 +05:30
Salvatore Sanfilippo ae1a9be8b0 Merge pull request #3324 from mishan/fix-wrong-comment-about-sentinel-mode
Fix incorrect comment for checkForSentinelMode function
2016-06-23 16:39:51 +02:00
antirez 67950680b9 Modules: mention RedisModule_Calloc() in the doc. 2016-06-23 16:20:48 +02:00
Salvatore Sanfilippo 8e9eef76bf Merge pull request #3335 from dvirsky/rm_calloc
added RM_Calloc implementation
2016-06-23 16:19:14 +02:00
antirez 2e9fd96c57 Merge branch 'unstable' of github.com:/antirez/redis into unstable 2016-06-23 16:18:57 +02:00
antirez b98fc4bfee Actually remove static from #3331.
I forgot -a when amending in the previous commit.
2016-06-23 16:18:30 +02:00
Salvatore Sanfilippo 9df3006aa8 Merge pull request #3336 from yossigo/create_string_from_string
Add RedisModule_CreateStringFromString().
2016-06-23 16:16:28 +02:00
antirez 3d42fa5b75 Minor change to conform PR #3331 to Redis code base style.
Also avoid "static" in order to have symbols during crashes.
2016-06-23 16:14:16 +02:00
Salvatore Sanfilippo f3b502da78 Merge pull request #3331 from yossigo/fix_openkey_crash
Fix occasional RM_OpenKey() crashes.
2016-06-23 16:12:07 +02:00
Salvatore Sanfilippo 18689db5f0 Merge pull request #3338 from tielei/unstable
A string with 21 chars is not representable as a 64-bit integer.
2016-06-23 16:06:15 +02:00
tielei 7b19bd3bb3 A string with 21 chars is not representable as a 64-bit integer. 2016-06-23 19:53:56 +08:00
Salvatore Sanfilippo c8cf3a4df6 Merge pull request #3330 from yossigo/fix_const
Use const in Redis Module API where possible.
2016-06-23 12:29:52 +02:00
antirez 4900e98d70 Modules: changes to logging function.
This commit changes what provided by PR #3315 (merged) in order to
let the user specify the log level as a string.

The define could be also used, but when this happens, they must be
decoupled from the defines in the Redis core, like in the other part of
the Redis modules implementations, so that a switch statement (or a
function) remaps between the two, otherwise we are no longer free to
change the internal Redis defines.
2016-06-23 12:11:30 +02:00
Yossi Gottlieb 84f1e03f9a Add RedisModule_Log() logging API function. 2016-06-23 12:01:44 +02:00
antirez 4b8cd3ab71 Commit change in autoMemoryFreed(): first -> last.
It's more natural to call the last entry added as "last", the original
commet got me confused until I actually read the code.
2016-06-23 09:38:30 +02:00
antirez 7bc37d864f Modules: implement zig-zag scanning in autoMemoryFreed().
Most of the time to check the last element is the way to go, however
there are patterns where the contrary is the best choice. Zig-zag
scanning implemented in this commmit always checks the obvious element
first (the last added -- think at a loop where the last element
allocated gets freed again and again), and continues checking one
element in the head and one in the tail.

Thanks to @dvisrky that fixed the original implementation of the
function and proposed zig zag scanning.
2016-06-23 09:09:51 +02:00
Salvatore Sanfilippo 574fc36d95 Merge pull request #3244 from dvirsky/optimize_autoMemoryFreed
Optimized autoMemoryFreed loop
2016-06-23 08:59:38 +02:00
Yossi Gottlieb 159c09f29e Add RedisModule_CreateStringFromString(). 2016-06-22 21:02:40 +03:00
Dvir Volk bdc079dca4 added RM_Calloc implementation 2016-06-22 17:32:41 +03:00
antirez aa345bf659 Modules doc: hint about replacing libc malloc calls. 2016-06-22 15:24:51 +02:00
Yossi Gottlieb 5b5b2f86c5 Cleanup: remove zset reset function from RM_ZsetRangeStop(). 2016-06-22 07:34:14 +03:00
Yossi Gottlieb fa96f33bef Fix occasional RM_OpenKey() crashes. 2016-06-21 10:22:19 +03:00
Yossi Gottlieb ce9d66d6e4 Use const in Redis Module API where possible. 2016-06-20 23:08:06 +03:00
Salvatore Sanfilippo ee9146d440 Merge pull request #3252 from oranagra/config_fix
fix: config set list-max-ziplist-size didn't support negative values
2016-06-17 14:48:41 +02:00
Misha Nasledov 42a2e12656 Fix incorrect comment for checkForSentinelMode function 2016-06-16 16:50:53 -07:00
antirez 9d450c4e7c Fix Sentinel pending commands counting.
This bug most experienced effect was an inability of Redis to
reconfigure back old masters to slaves after they are reachable again
after a failover. This was due to failing to reset the count of the
pending commands properly, so the master appeared fovever down.

Was introduced in Redis 3.2 new Sentinel connection sharing feature
which is a lot more complex than the 3.0 code, but more scalable.

Many thanks to people reporting the issue, and especially to
@sskorgal for investigating the issue in depth.

Hopefully closes #3285.
2016-06-16 19:27:24 +02:00
antirez 1f0bd01302 redis-cli: really connect to the right server.
I recently introduced populating the autocomplete help array with the
COMMAND command if available. However this was performed before parsing
the arguments, defaulting to instance 6379. After the connection is
performed it remains stable.

The effect is that if there is an instance running on port 6339,
whatever port you specify is ignored and 6379 is connected to instead.
The right port will be selected only after a reconnection.

Close #3314.
2016-06-16 17:23:31 +02:00
antirez 975c5bc7fe RESTORE: accept RDB dumps with older versions.
Reference issue #3218.

Checking the code I can't find a reason why the original RESTORE
code was so opinionated about restoring only the current version. The
code in to `rdb.c` appears to be capable as always to restore data from
older versions of Redis, and the only places where it is needed the
current version in order to correctly restore data, is while loading the
opcodes, not the values itself as it happens in the case of RESTORE.

For the above reasons, this commit enables RESTORE to accept older
versions of values payloads.
2016-06-16 15:53:57 +02:00
Salvatore Sanfilippo b869f91532 Merge pull request #3255 from oranagra/error_string
CLIENT error message was out of date
2016-06-16 12:59:05 +02:00
Salvatore Sanfilippo 3041842245 Merge pull request #3256 from oranagra/georasius_neg
fix georadius returns multiple replies
2016-06-16 12:57:59 +02:00
antirez 4f2bf52409 Minor aesthetic fixes to PR #3264.
Comment format fixed + local var modified from camel case to underscore
separators as Redis code base normally does (camel case is mostly used
for global symbols like structure names, function names, global vars,
...).
2016-06-16 12:54:33 +02:00
Salvatore Sanfilippo fbb8fcbb76 Merge pull request #3264 from oranagra/bitfield_fix2
fix crash in BITFIELD GET on non existing key or wrong type see #3259
2016-06-16 12:52:36 +02:00
Salvatore Sanfilippo 1fcac96abe Merge pull request #3274 from MOON-CLJ/fix_promoted_slave
Sentinel: fix check when can't send the command to the promoted slave
2016-06-15 17:24:11 +02:00
antirez 343b54cdbe Test TOUCH and new TTL / TYPE behavior about object access time. 2016-06-15 17:15:51 +02:00
Salvatore Sanfilippo 5123bc2e69 Merge pull request #3283 from ideal/unstable
fix mistake in comment in object.c
2016-06-15 15:53:19 +02:00
Salvatore Sanfilippo c533c5c8bc Merge pull request #3281 from jamespedwards42/unstable
Fix modules intro typos.
2016-06-15 12:51:15 +02:00
jamespedwards42 1bd8a08435 Fix modules intro typos. 2016-05-29 15:53:24 -07:00
643 changed files with 144865 additions and 8558 deletions

No files matched your search

+8
View File
@@ -1,14 +1,22 @@
.*.swp
core
*.o
*.log
dump.rdb
redis-benchmark
keydb-benchmark
redis-check-aof
keydb-check-aof
redis-check-rdb
keydb-check-rdb
redis-check-dump
keydb-check-dump
redis-cli
keydb-cli
redis-sentinel
keydb-sentinel
redis-server
keydb-server
doc-tools
release
misc/*
+28
View File
@@ -0,0 +1,28 @@
language: generic
os: osx
matrix:
include:
- os: linux
script: make
env: COMPILER_NAME=gcc CXX=g++-5 CC=gcc-5
addons:
apt:
packages:
- g++-5
- nasm
- uuid-dev
sources: &sources
- llvm-toolchain-precise-3.8
- ubuntu-toolchain-r-test
- os: linux
script: make MALLOC=libc
env: COMPILER_NAME=clang CXX=clang++-3.8 CC=clang-3.8 CXXFLAGS="-stdlib=libc++" LDFLAGS="-stdlib=libc++"
addons:
apt:
packages:
- clang-3.8
- libc++-dev
- libc++abi-dev
- nasm
- uuid-dev
sources: *sources
+56
View File
@@ -0,0 +1,56 @@
{
"files.associations": {
"zmalloc.h": "c",
"stat.h": "c",
"array": "cpp",
"atomic": "cpp",
"*.tcc": "cpp",
"cctype": "cpp",
"chrono": "cpp",
"clocale": "cpp",
"cmath": "cpp",
"condition_variable": "cpp",
"cstdarg": "cpp",
"cstddef": "cpp",
"cstdint": "cpp",
"cstdio": "cpp",
"cstdlib": "cpp",
"cstring": "cpp",
"ctime": "cpp",
"cwchar": "cpp",
"cwctype": "cpp",
"deque": "cpp",
"list": "cpp",
"unordered_map": "cpp",
"vector": "cpp",
"exception": "cpp",
"fstream": "cpp",
"functional": "cpp",
"future": "cpp",
"initializer_list": "cpp",
"iomanip": "cpp",
"iosfwd": "cpp",
"iostream": "cpp",
"istream": "cpp",
"limits": "cpp",
"memory": "cpp",
"mutex": "cpp",
"new": "cpp",
"numeric": "cpp",
"optional": "cpp",
"ostream": "cpp",
"ratio": "cpp",
"scoped_allocator": "cpp",
"sstream": "cpp",
"stdexcept": "cpp",
"streambuf": "cpp",
"string_view": "cpp",
"system_error": "cpp",
"thread": "cpp",
"cinttypes": "cpp",
"tuple": "cpp",
"type_traits": "cpp",
"typeinfo": "cpp",
"utility": "cpp"
}
}
+1
View File
@@ -1,4 +1,5 @@
Copyright (c) 2006-2015, Salvatore Sanfilippo
Copyright (C) 2019, John Sully
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+10
View File
@@ -0,0 +1,10 @@
FROM ubuntu:18.04
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -qqy \
build-essential nasm autotools-dev autoconf libjemalloc-dev tcl tcl-dev uuid-dev \
&& apt-get clean
CMD make
-106
View File
@@ -1,106 +0,0 @@
[Note: this is the Redis manifesto, for general information about
installing and running Redis read the README file instead.]
Redis Manifesto
===============
1 - A DSL for Abstract Data Types. Redis is a DSL (Domain Specific Language)
that manipulates abstract data types and implemented as a TCP daemon.
Commands manipulate a key space where keys are binary-safe strings and
values are different kinds of abstract data types. Every data type
represents an abstract version of a fundamental data structure. For instance
Redis Lists are an abstract representation of linked lists. In Redis, the
essence of a data type isn't just the kind of operations that the data types
support, but also the space and time complexity of the data type and the
operations performed upon it.
2 - Memory storage is #1. The Redis data set, composed of defined key-value
pairs, is primarily stored in the computer's memory. The amount of memory in
all kinds of computers, including entry-level servers, is increasing
significantly each year. Memory is fast, and allows Redis to have very
predictable performance. Datasets composed of 10k or 40 millions keys will
perform similarly. Complex data types like Redis Sorted Sets are easy to
implement and manipulate in memory with good performance, making Redis very
simple. Redis will continue to explore alternative options (where data can
be optionally stored on disk, say) but the main goal of the project remains
the development of an in-memory database.
3 - Fundamental data structures for a fundamental API. The Redis API is a direct
consequence of fundamental data structures. APIs can often be arbitrary but
not an API that resembles the nature of fundamental data structures. If we
ever meet intelligent life forms from another part of the universe, they'll
likely know, understand and recognize the same basic data structures we have
in our computer science books. Redis will avoid intermediate layers in API,
so that the complexity is obvious and more complex operations can be
performed as the sum of the basic operations.
4 - We believe in code efficiency. Computers get faster and faster, yet we
believe that abusing computing capabilities is not wise: the amount of
operations you can do for a given amount of energy remains anyway a
significant parameter: it allows to do more with less computers and, at
the same time, having a smaller environmental impact. Similarly Redis is
able to "scale down" to smaller devices. It is perfectly usable in a
Raspberry Pi and other small ARM based computers. Faster code having
just the layers of abstractions that are really needed will also result,
often, in more predictable performances. We think likewise about memory
usage, one of the fundamental goals of the Redis project is to
incrementally build more and more memory efficient data structures, so that
problems that were not approachable in RAM in the past will be perfectly
fine to handle in the future.
5 - Code is like a poem; it's not just something we write to reach some
practical result. Sometimes people that are far from the Redis philosophy
suggest using other code written by other authors (frequently in other
languages) in order to implement something Redis currently lacks. But to us
this is like if Shakespeare decided to end Enrico IV using the Paradiso from
the Divina Commedia. Is using any external code a bad idea? Not at all. Like
in "One Thousand and One Nights" smaller self contained stories are embedded
in a bigger story, we'll be happy to use beautiful self contained libraries
when needed. At the same time, when writing the Redis story we're trying to
write smaller stories that will fit in to other code.
6 - We're against complexity. We believe designing systems is a fight against
complexity. We'll accept to fight the complexity when it's worthwhile but
we'll try hard to recognize when a small feature is not worth 1000s of lines
of code. Most of the time the best way to fight complexity is by not
creating it at all. Complexity is also a form of lock-in: code that is
very hard to understand cannot be modified by users in an independent way
regardless of the license. One of the main Redis goals is to remain
understandable, enough for a single programmer to have a clear idea of how
it works in detail just reading the source code for a couple of weeks.
7 - Threading is not a silver bullet. Instead of making Redis threaded we
believe on the idea of an efficient (mostly) single threaded Redis core.
Multiple of such cores, that may run in the same computer or may run
in multiple computers, are abstracted away as a single big system by
higher order protocols and features: Redis Cluster and the upcoming
Redis Proxy are our main goals. A shared nothing approach is not just
much simpler (see the previous point in this document), is also optimal
in NUMA systems. In the specific case of Redis it allows for each instance
to have a more limited amount of data, making the Redis persist-by-fork
approach more sounding. In the future we may explore parallelism only for
I/O, which is the low hanging fruit: minimal complexity could provide an
improved single process experience.
8 - Two levels of API. The Redis API has two levels: 1) a subset of the API fits
naturally into a distributed version of Redis and 2) a more complex API that
supports multi-key operations. Both are useful if used judiciously but
there's no way to make the more complex multi-keys API distributed in an
opaque way without violating our other principles. We don't want to provide
the illusion of something that will work magically when actually it can't in
all cases. Instead we'll provide commands to quickly migrate keys from one
instance to another to perform multi-key operations and expose the
trade-offs to the user.
9 - We optimize for joy. We believe writing code is a lot of hard work, and the
only way it can be worth is by enjoying it. When there is no longer joy in
writing code, the best thing to do is stop. To prevent this, we'll avoid
taking paths that will make Redis less of a joy to develop.
10 - All the above points are put together in what we call opportunistic
programming: trying to get the most for the user with minimal increases
in complexity (hanging fruits). Solve 95% of the problem with 5% of the
code when it is acceptable. Avoid a fixed schedule but follow the flow of
user requests, inspiration, Redis internal readiness for certain features
(sometimes many past changes reach a critical point making a previously
complex feature very easy to obtain).
+123 -319
View File
@@ -1,52 +1,83 @@
This README is just a fast *quick start* document. You can find more detailed documentation at [redis.io](https://redis.io).
![Current Release](https://img.shields.io/github/release/JohnSully/KeyDB.svg)
[![Build Status](https://travis-ci.org/JohnSully/KeyDB.svg?branch=unstable)](https://travis-ci.org/JohnSully/KeyDB) [![Join the chat at https://gitter.im/KeyDB/community](https://badges.gitter.im/KeyDB/community.svg)](https://gitter.im/KeyDB/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
What is Redis?
What is KeyDB?
--------------
Redis is often referred as a *data structures* server. What this means is that Redis provides access to mutable data structures via a set of commands, which are sent using a *server-client* model with TCP sockets and a simple protocol. So different processes can query and modify the same data structures in a shared way.
KeyDB is a high performance fork of Redis focusing on multithreading, memory efficiency, and high throughput. In addition to multithreading KeyDB also has features only available in Redis Enterprise such as [Active Replication](https://github.com/JohnSully/KeyDB/wiki/KeyDB-(Redis-Fork):-Active-Replica-Support), [FLASH storage](https://github.com/JohnSully/KeyDB/wiki/FLASH-Storage) support, and some not available at all such as direct backup to AWS S3.
Data structures implemented into Redis have a few special properties:
On the same hardware KeyDB can perform twice as many queries per second as Redis, with 60% lower latency.
* Redis cares to store them on disk, even if they are always served and modified into the server memory. This means that Redis is fast, but that is also non-volatile.
* Implementation of data structures stress on memory efficiency, so data structures inside Redis will likely use less memory compared to the same data structure modeled using an high level programming language.
* Redis offers a number of features that are natural to find in a database, like replication, tunable levels of durability, cluster, high availability.
KeyDB has full compatibility with the Redis protocol, modules, and scripts. This includes full support for transactions, and atomic execution of scripts. For more information see our architecture section below.
Another good example is to think of Redis as a more complex version of memcached, where the operations are not just SETs and GETs, but operations to work with complex data types like Lists, Sets, ordered data structures, and so forth.
Try our docker container: https://hub.docker.com/r/eqalpha/keydb
If you want to know more, this is a list of selected starting points:
Talk on Gitter: https://gitter.im/KeyDB
* Introduction to Redis data types. http://redis.io/topics/data-types-intro
* Try Redis directly inside your browser. http://try.redis.io
* The full list of Redis commands. http://redis.io/commands
* There is much more inside the Redis official documentation. http://redis.io/documentation
Management GUI: We recommend [FastoNoSQL](https://fastonosql.com/) which has official KeyDB support.
Building Redis
New: Active Replica Support
---------------------------
New! KeyDB now has support for Active Replicas. This feature greatly simplifies hot-spare failover and allows you to distribute writes over replicas instead of just a single master. For more information [see the wiki page](https://github.com/JohnSully/KeyDB/wiki/KeyDB-(Redis-Fork):-Active-Replica-Support).
Why fork Redis?
---------------
The Redis maintainers have continually reiterated that they do not plan to support multithreading. While we have great respect for the redis team, we feel the analysis justifying this decision is incorrect. In addition we wanted open source implementations of features currently only available in proprietary modules. We feel a fork is the best way to accelerate development in the areas of most interest to us.
We plan to track the Redis repo closely and hope our projects can learn from each other.
Benchmarking KeyDB
------------------
<img src=https://cdn-images-1.medium.com/max/1400/1*s7mTb7Qb0kxc951mz8bdgA.png width=420 height=300/><img src=https://cdn-images-1.medium.com/max/1400/1*R00A5U4AFGohGOYHMfT6fA.png height=300/>
Please note keydb-benchmark and redis-benchmark are currently single threaded and too slow to properly benchmark KeyDB. We recommend using a redis cluster benchmark tool such as [memtier](https://github.com/RedisLabs/memtier_benchmark). Please ensure your machine has enough cores for both KeyDB and memteir if testing locally. KeyDB expects exclusive use of any cores assigned to it.
For more details on how we benchmarked KeyDB along with performance numbers check out our blog post: [Redis Should Be Multithreaded](https://medium.com/@john_63123/redis-should-be-multi-threaded-e28319cab744?source=friends_link&sk=7ce8e9fe3ec8224a4d27ef075d085457)
New Configuration Options
-------------------------
With new features comes new options:
server-threads N
server-thread-affinity [true/false]
The number of threads used to serve requests. This should be related to the number of queues available in your network hardware, *not* the number of cores on your machine. Because KeyDB uses spinlocks to reduce latency; making this too high will reduce performance. We recommend using 4 here. By default this is set to one.
scratch-file-path /path
If you would like to use the [FLASH backed](https://github.com/JohnSully/KeyDB/wiki/FLASH-Storage) storage this option configures the directory for KeyDB's temporary files. This feature relies on snapshotting to work so must be used on a BTRFS filesystem. ZFS may also work but is untested. With this feature KeyDB will use RAM as a cache and page to disk as necessary. NOTE: This requires special compilation options, see Building KeyDB below.
db-s3-object /path/to/bucket
If you would like KeyDB to dump and load directly to AWS S3 this option specifies the bucket. Using this option with the traditional RDB options will result in KeyDB backing up twice to both locations. If both are specified KeyDB will first attempt to load from the local dump file and if that fails load from S3. This requires the AWS CLI tools to be installed and configured which are used under the hood to transfer the data.
All other configuration options behave as you'd expect. Your existing configuration files should continue to work unchanged.
Building KeyDB
--------------
Redis can be compiled and used on Linux, OSX, OpenBSD, NetBSD, FreeBSD.
We support big endian and little endian architectures, and both 32 bit
and 64 bit systems.
KeyDB can be compiled and is tested for use on Linux. KeyDB currently relies on SO_REUSEPORT's load balancing behavior which is available only in Linux. When we support marshalling connections across threads we plan to support other operating systems such as FreeBSD.
It may compile on Solaris derived systems (for instance SmartOS) but our
support for this platform is *best effort* and Redis is not guaranteed to
work as well as in Linux, OSX, and \*BSD there.
Install dependencies:
It is as simple as:
% sudo apt install build-essential nasm autotools-dev autoconf libjemalloc-dev tcl tcl-dev uuid-dev
Compiling is as simple as:
% make
You can run a 32 bit Redis binary using:
You can enable flash support with (Note: autoconf and autotools must be installed):
% make 32bit
After building Redis, it is a good idea to test it using:
% make test
% make MALLOC=memkind
Fixing build problems with dependencies or cached build options
---------
Redis has some dependencies which are included into the `deps` directory.
KeyDB has some dependencies which are included into the `deps` directory.
`make` does not automatically rebuild dependencies even if something in
the source code of dependencies changes.
@@ -66,11 +97,11 @@ command.
Fixing problems building 32 bit binaries
---------
If after building Redis with a 32 bit target you need to rebuild it
If after building KeyDB with a 32 bit target you need to rebuild it
with a 64 bit target, or the other way around, you need to perform a
`make distclean` in the root directory of the Redis distribution.
`make distclean` in the root directory of the KeyDB distribution.
In case of build errors when trying to build a 32 bit binary of Redis, try
In case of build errors when trying to build a 32 bit binary of KeyDB, try
the following steps:
* Install the packages libc6-dev-i386 (also try g++-multilib).
@@ -80,8 +111,8 @@ the following steps:
Allocator
---------
Selecting a non-default memory allocator when building Redis is done by setting
the `MALLOC` environment variable. Redis is compiled and linked against libc
Selecting a non-default memory allocator when building KeyDB is done by setting
the `MALLOC` environment variable. KeyDB is compiled and linked against libc
malloc by default, with the exception of jemalloc being the default on Linux
systems. This default was picked because jemalloc has proven to have fewer
fragmentation problems than libc malloc.
@@ -97,60 +128,60 @@ To compile against jemalloc on Mac OS X systems, use:
Verbose build
-------------
Redis will build with a user friendly colorized output by default.
KeyDB will build with a user friendly colorized output by default.
If you want to see a more verbose output use the following:
% make V=1
Running Redis
Running KeyDB
-------------
To run Redis with the default configuration just type:
To run KeyDB with the default configuration just type:
% cd src
% ./redis-server
% ./keydb-server
If you want to provide your redis.conf, you have to run it using an additional
If you want to provide your keydb.conf, you have to run it using an additional
parameter (the path of the configuration file):
% cd src
% ./redis-server /path/to/redis.conf
% ./keydb-server /path/to/keydb.conf
It is possible to alter the Redis configuration by passing parameters directly
It is possible to alter the KeyDB configuration by passing parameters directly
as options using the command line. Examples:
% ./redis-server --port 9999 --replicaof 127.0.0.1 6379
% ./redis-server /etc/redis/6379.conf --loglevel debug
% ./keydb-server --port 9999 --replicaof 127.0.0.1 6379
% ./keydb-server /etc/keydb/6379.conf --loglevel debug
All the options in redis.conf are also supported as options using the command
All the options in keydb.conf are also supported as options using the command
line, with exactly the same name.
Playing with Redis
Playing with KeyDB
------------------
You can use redis-cli to play with Redis. Start a redis-server instance,
You can use keydb-cli to play with KeyDB. Start a keydb-server instance,
then in another terminal try the following:
% cd src
% ./redis-cli
redis> ping
% ./keydb-cli
keydb> ping
PONG
redis> set foo bar
keydb> set foo bar
OK
redis> get foo
keydb> get foo
"bar"
redis> incr mycounter
keydb> incr mycounter
(integer) 1
redis> incr mycounter
keydb> incr mycounter
(integer) 2
redis>
keydb>
You can find the list of all the available commands at http://redis.io/commands.
Installing Redis
Installing KeyDB
-----------------
In order to install Redis binaries into /usr/local/bin just use:
In order to install KeyDB binaries into /usr/local/bin just use:
% make install
@@ -159,7 +190,7 @@ different destination.
Make install will just install binaries in your system, but will not configure
init scripts and configuration files in the appropriate place. This is not
needed if you want just to play a bit with Redis, but if you are installing
needed if you want just to play a bit with KeyDB, but if you are installing
it the proper way for a production system, we have a script doing this
for Ubuntu and Debian systems:
@@ -169,280 +200,53 @@ for Ubuntu and Debian systems:
_Note_: `install_server.sh` will not work on Mac OSX; it is built for Linux only.
The script will ask you a few questions and will setup everything you need
to run Redis properly as a background daemon that will start again on
to run KeyDB properly as a background daemon that will start again on
system reboots.
You'll be able to stop and start Redis using the script named
`/etc/init.d/redis_<portnumber>`, for instance `/etc/init.d/redis_6379`.
You'll be able to stop and start KeyDB using the script named
`/etc/init.d/keydb_<portnumber>`, for instance `/etc/init.d/keydb_6379`.
Multithreading Architecture
---------------------------
KeyDB works by running the normal Redis event loop on multiple threads. Network IO, and query parsing are done concurrently. Each connection is assigned a thread on accept(). Access to the core hash table is guarded by spinlock. Because the hashtable access is extremely fast this lock has low contention. Transactions hold the lock for the duration of the EXEC command. Modules work in concert with the GIL which is only acquired when all server threads are paused. This maintains the atomicity guarantees modules expect.
Unlike most databases the core data structure is the fastest part of the system. Most of the query time comes from parsing the REPL protocol and copying data to/from the network.
Future work:
- Allow rebalancing of connections to different threads after the connection
- Allow multiple readers access to the hashtable concurrently
Docker Build
------------
Run the following commands for a full source download and build:
```
git clone git@github.com:JohnSully/KeyDB.git
docker run -it --rm -v `pwd`/KeyDB:/build -w /build devopsdood/keydb-builder make
```
Then you have fresh binaries built, you can also pass any other options to the make command above after the word make. E.g.
```docker run -it --rm -v `pwd`/KeyDB:/build -w /build devopsdood/keydb-builder make MAllOC=memkind```
The above commands will build you binaries in the src directory. Standard `make install` without Docker command will work after if you wish to install
If you'd prefer you can build the Dockerfile in the repo instead of pulling the above container for use:
`docker build -t KeyDB .`
Code contributions
-----------------
Note: by contributing code to the Redis project in any form, including sending
Note: by contributing code to the KeyDB project in any form, including sending
a pull request via Github, a code fragment or patch via private email or
public discussion groups, you agree to release your code under the terms
of the BSD license that you can find in the [COPYING][1] file included in the Redis
of the BSD license that you can find in the COPYING file included in the KeyDB
source distribution.
Please see the [CONTRIBUTING][2] file in this source distribution for more
Please see the CONTRIBUTING file in this source distribution for more
information.
[1]: https://github.com/antirez/redis/blob/unstable/COPYING
[2]: https://github.com/antirez/redis/blob/unstable/CONTRIBUTING
Redis internals
===
If you are reading this README you are likely in front of a Github page
or you just untarred the Redis distribution tar ball. In both the cases
you are basically one step away from the source code, so here we explain
the Redis source code layout, what is in each file as a general idea, the
most important functions and structures inside the Redis server and so forth.
We keep all the discussion at a high level without digging into the details
since this document would be huge otherwise and our code base changes
continuously, but a general idea should be a good starting point to
understand more. Moreover most of the code is heavily commented and easy
to follow.
Source code layout
---
The Redis root directory just contains this README, the Makefile which
calls the real Makefile inside the `src` directory and an example
configuration for Redis and Sentinel. You can find a few shell
scripts that are used in order to execute the Redis, Redis Cluster and
Redis Sentinel unit tests, which are implemented inside the `tests`
directory.
Inside the root are the following important directories:
* `src`: contains the Redis implementation, written in C.
* `tests`: contains the unit tests, implemented in Tcl.
* `deps`: contains libraries Redis uses. Everything needed to compile Redis is inside this directory; your system just needs to provide `libc`, a POSIX compatible interface and a C compiler. Notably `deps` contains a copy of `jemalloc`, which is the default allocator of Redis under Linux. Note that under `deps` there are also things which started with the Redis project, but for which the main repository is not `antirez/redis`.
There are a few more directories but they are not very important for our goals
here. We'll focus mostly on `src`, where the Redis implementation is contained,
exploring what there is inside each file. The order in which files are
exposed is the logical one to follow in order to disclose different layers
of complexity incrementally.
Note: lately Redis was refactored quite a bit. Function names and file
names have been changed, so you may find that this documentation reflects the
`unstable` branch more closely. For instance in Redis 3.0 the `server.c`
and `server.h` files were named `redis.c` and `redis.h`. However the overall
structure is the same. Keep in mind that all the new developments and pull
requests should be performed against the `unstable` branch.
server.h
---
The simplest way to understand how a program works is to understand the
data structures it uses. So we'll start from the main header file of
Redis, which is `server.h`.
All the server configuration and in general all the shared state is
defined in a global structure called `server`, of type `struct redisServer`.
A few important fields in this structure are:
* `server.db` is an array of Redis databases, where data is stored.
* `server.commands` is the command table.
* `server.clients` is a linked list of clients connected to the server.
* `server.master` is a special client, the master, if the instance is a replica.
There are tons of other fields. Most fields are commented directly inside
the structure definition.
Another important Redis data structure is the one defining a client.
In the past it was called `redisClient`, now just `client`. The structure
has many fields, here we'll just show the main ones:
struct client {
int fd;
sds querybuf;
int argc;
robj **argv;
redisDb *db;
int flags;
list *reply;
char buf[PROTO_REPLY_CHUNK_BYTES];
... many other fields ...
}
The client structure defines a *connected client*:
* The `fd` field is the client socket file descriptor.
* `argc` and `argv` are populated with the command the client is executing, so that functions implementing a given Redis command can read the arguments.
* `querybuf` accumulates the requests from the client, which are parsed by the Redis server according to the Redis protocol and executed by calling the implementations of the commands the client is executing.
* `reply` and `buf` are dynamic and static buffers that accumulate the replies the server sends to the client. These buffers are incrementally written to the socket as soon as the file descriptor is writable.
As you can see in the client structure above, arguments in a command
are described as `robj` structures. The following is the full `robj`
structure, which defines a *Redis object*:
typedef struct redisObject {
unsigned type:4;
unsigned encoding:4;
unsigned lru:LRU_BITS; /* lru time (relative to server.lruclock) */
int refcount;
void *ptr;
} robj;
Basically this structure can represent all the basic Redis data types like
strings, lists, sets, sorted sets and so forth. The interesting thing is that
it has a `type` field, so that it is possible to know what type a given
object has, and a `refcount`, so that the same object can be referenced
in multiple places without allocating it multiple times. Finally the `ptr`
field points to the actual representation of the object, which might vary
even for the same type, depending on the `encoding` used.
Redis objects are used extensively in the Redis internals, however in order
to avoid the overhead of indirect accesses, recently in many places
we just use plain dynamic strings not wrapped inside a Redis object.
server.c
---
This is the entry point of the Redis server, where the `main()` function
is defined. The following are the most important steps in order to startup
the Redis server.
* `initServerConfig()` setups the default values of the `server` structure.
* `initServer()` allocates the data structures needed to operate, setup the listening socket, and so forth.
* `aeMain()` starts the event loop which listens for new connections.
There are two special functions called periodically by the event loop:
1. `serverCron()` is called periodically (according to `server.hz` frequency), and performs tasks that must be performed from time to time, like checking for timedout clients.
2. `beforeSleep()` is called every time the event loop fired, Redis served a few requests, and is returning back into the event loop.
Inside server.c you can find code that handles other vital things of the Redis server:
* `call()` is used in order to call a given command in the context of a given client.
* `activeExpireCycle()` handles eviciton of keys with a time to live set via the `EXPIRE` command.
* `freeMemoryIfNeeded()` is called when a new write command should be performed but Redis is out of memory according to the `maxmemory` directive.
* The global variable `redisCommandTable` defines all the Redis commands, specifying the name of the command, the function implementing the command, the number of arguments required, and other properties of each command.
networking.c
---
This file defines all the I/O functions with clients, masters and replicas
(which in Redis are just special clients):
* `createClient()` allocates and initializes a new client.
* the `addReply*()` family of functions are used by commands implementations in order to append data to the client structure, that will be transmitted to the client as a reply for a given command executed.
* `writeToClient()` transmits the data pending in the output buffers to the client and is called by the *writable event handler* `sendReplyToClient()`.
* `readQueryFromClient()` is the *readable event handler* and accumulates data from read from the client into the query buffer.
* `processInputBuffer()` is the entry point in order to parse the client query buffer according to the Redis protocol. Once commands are ready to be processed, it calls `processCommand()` which is defined inside `server.c` in order to actually execute the command.
* `freeClient()` deallocates, disconnects and removes a client.
aof.c and rdb.c
---
As you can guess from the names these files implement the RDB and AOF
persistence for Redis. Redis uses a persistence model based on the `fork()`
system call in order to create a thread with the same (shared) memory
content of the main Redis thread. This secondary thread dumps the content
of the memory on disk. This is used by `rdb.c` to create the snapshots
on disk and by `aof.c` in order to perform the AOF rewrite when the
append only file gets too big.
The implementation inside `aof.c` has additional functions in order to
implement an API that allows commands to append new commands into the AOF
file as clients execute them.
The `call()` function defined inside `server.c` is responsible to call
the functions that in turn will write the commands into the AOF.
db.c
---
Certain Redis commands operate on specific data types, others are general.
Examples of generic commands are `DEL` and `EXPIRE`. They operate on keys
and not on their values specifically. All those generic commands are
defined inside `db.c`.
Moreover `db.c` implements an API in order to perform certain operations
on the Redis dataset without directly accessing the internal data structures.
The most important functions inside `db.c` which are used in many commands
implementations are the following:
* `lookupKeyRead()` and `lookupKeyWrite()` are used in order to get a pointer to the value associated to a given key, or `NULL` if the key does not exist.
* `dbAdd()` and its higher level counterpart `setKey()` create a new key in a Redis database.
* `dbDelete()` removes a key and its associated value.
* `emptyDb()` removes an entire single database or all the databases defined.
The rest of the file implements the generic commands exposed to the client.
object.c
---
The `robj` structure defining Redis objects was already described. Inside
`object.c` there are all the functions that operate with Redis objects at
a basic level, like functions to allocate new objects, handle the reference
counting and so forth. Notable functions inside this file:
* `incrRefcount()` and `decrRefCount()` are used in order to increment or decrement an object reference count. When it drops to 0 the object is finally freed.
* `createObject()` allocates a new object. There are also specialized functions to allocate string objects having a specific content, like `createStringObjectFromLongLong()` and similar functions.
This file also implements the `OBJECT` command.
replication.c
---
This is one of the most complex files inside Redis, it is recommended to
approach it only after getting a bit familiar with the rest of the code base.
In this file there is the implementation of both the master and replica role
of Redis.
One of the most important functions inside this file is `replicationFeedSlaves()` that writes commands to the clients representing replica instances connected
to our master, so that the replicas can get the writes performed by the clients:
this way their data set will remain synchronized with the one in the master.
This file also implements both the `SYNC` and `PSYNC` commands that are
used in order to perform the first synchronization between masters and
replicas, or to continue the replication after a disconnection.
Other C files
---
* `t_hash.c`, `t_list.c`, `t_set.c`, `t_string.c` and `t_zset.c` contains the implementation of the Redis data types. They implement both an API to access a given data type, and the client commands implementations for these data types.
* `ae.c` implements the Redis event loop, it's a self contained library which is simple to read and understand.
* `sds.c` is the Redis string library, check http://github.com/antirez/sds for more information.
* `anet.c` is a library to use POSIX networking in a simpler way compared to the raw interface exposed by the kernel.
* `dict.c` is an implementation of a non-blocking hash table which rehashes incrementally.
* `scripting.c` implements Lua scripting. It is completely self contained from the rest of the Redis implementation and is simple enough to understand if you are familar with the Lua API.
* `cluster.c` implements the Redis Cluster. Probably a good read only after being very familiar with the rest of the Redis code base. If you want to read `cluster.c` make sure to read the [Redis Cluster specification][3].
[3]: http://redis.io/topics/cluster-spec
Anatomy of a Redis command
---
All the Redis commands are defined in the following way:
void foobarCommand(client *c) {
printf("%s",c->argv[1]->ptr); /* Do something with the argument. */
addReply(c,shared.ok); /* Reply something to the client. */
}
The command is then referenced inside `server.c` in the command table:
{"foobar",foobarCommand,2,"rtF",0,NULL,0,0,0,0,0},
In the above example `2` is the number of arguments the command takes,
while `"rtF"` are the command flags, as documented in the command table
top comment inside `server.c`.
After the command operates in some way, it returns a reply to the client,
usually using `addReply()` or a similar function defined inside `networking.c`.
There are tons of commands implementations inside the Redis source code
that can serve as examples of actual commands implementations. To write
a few toy commands can be a good exercise to familiarize with the code base.
There are also many other files not described here, but it is useless to
cover everything. We want to just help you with the first steps.
Eventually you'll find your way inside the Redis code base :-)
Enjoy!
+5
View File
@@ -53,6 +53,11 @@ linenoise: .make-prerequisites
.PHONY: linenoise
.PHONY: memkind
memkind:
cd memkind && $(MAKE)
ifeq ($(uname_S),SunOS)
# Make isinf() available
LUA_CFLAGS= -D__C99FEATURES__=1
+2 -2
View File
@@ -23,9 +23,9 @@ INSTALL_INCLUDE_PATH= $(DESTDIR)$(PREFIX)/$(INCLUDE_PATH)
INSTALL_LIBRARY_PATH= $(DESTDIR)$(PREFIX)/$(LIBRARY_PATH)
INSTALL_PKGCONF_PATH= $(INSTALL_LIBRARY_PATH)/$(PKGCONF_PATH)
# redis-server configuration used for testing
# keydb-server configuration used for testing
REDIS_PORT=56379
REDIS_SERVER=redis-server
REDIS_SERVER=keydb-server
define REDIS_TEST_CONFIG
daemonize yes
pidfile /tmp/hiredis-test-redis.pid
+12 -1
View File
@@ -45,31 +45,41 @@ typedef char *sds;
* However is here to document the layout of type 5 SDS strings. */
struct __attribute__ ((__packed__)) sdshdr5 {
unsigned char flags; /* 3 lsb of type, and 5 msb of string length */
#ifndef __cplusplus
char buf[];
#endif
};
struct __attribute__ ((__packed__)) sdshdr8 {
uint8_t len; /* used */
uint8_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
#ifndef __cplusplus
char buf[];
#endif
};
struct __attribute__ ((__packed__)) sdshdr16 {
uint16_t len; /* used */
uint16_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
#ifndef __cplusplus
char buf[];
#endif
};
struct __attribute__ ((__packed__)) sdshdr32 {
uint32_t len; /* used */
uint32_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
#ifndef __cplusplus
char buf[];
#endif
};
struct __attribute__ ((__packed__)) sdshdr64 {
uint64_t len; /* used */
uint64_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
#ifndef __cplusplus
char buf[];
#endif
};
#define SDS_TYPE_5 0
@@ -85,7 +95,8 @@ struct __attribute__ ((__packed__)) sdshdr64 {
static inline size_t sdslen(const sds s) {
unsigned char flags = s[-1];
switch(flags&SDS_TYPE_MASK) {
switch(__builtin_expect((flags&SDS_TYPE_MASK), SDS_TYPE_5)) {
case SDS_TYPE_5:
return SDS_TYPE_5_LEN(flags);
case SDS_TYPE_8:
+12
View File
@@ -0,0 +1,12 @@
export JE_PREFIX=jemk_
.PHONY: all
all:
cd src; ./build_jemalloc.sh
cd src; ./build.sh
.PHONY: clean
clean:
cd src; make clean
+113
View File
@@ -0,0 +1,113 @@
.cproject
.project
autom4te.cache/
*.a
*.o
*.so
*.la
*.lo
.deps/
.dirstamp
.libs/
/aclocal.m4
/ar-lib
/compile
/config.guess
/config.h
/config.h.in
/config.log
/config.status
/config.sub
/configure
/depcomp
/config_tls.h
/install-sh
/libtool
/ltmain.sh
/m4/
/memkind-*.spec
/memkind-*.tar.gz
/Makefile
/Makefile.in
/missing
/stamp-h1
/stamp-h2
/jemalloc/obj/
/jemalloc/configure
/src/memkind_defines.h.in
/test/all_tests
/test/.deps/
/test/.dirstamp
/test/.libs/
/test/*.o
/test/performance/.deps/
/test/*.so
/test/check.sh.log
/test/check.sh.trs
/test-suite.log
/VERSION
/examples/.deps/
/examples/.dirstamp
/examples/.libs/
/examples/*.o
/examples/*.lo
/examples/*.la
/memkind-hbw-nodes
/test/allocator_perf_tool_tests
/test/autohbw_candidates
/test/locality_test
/test/decorator_test
/test/eratosthenes
/test/environerr_hbw_malloc_test
/test/environerr_test
/test/freeing_memory_segfault_test
/test/filter_memkind
/test/gb_page_tests_bind_policy
/test/hello_hbw
/test/hello_memkind
/test/hello_memkind_debug
/test/mallocerr_test
/test/memkind_allocated
/test/new_kind
/test/perf_tool
/test/pmem_kinds
/test/pmem_malloc
/test/pmem_malloc_unlimited
/test/pmem_usable_size
/test/pmem_alignment
/test/pmem_and_default_kind
/test/pmem_multithreads
/test/pmem_multithreads_onekind
/test/pmem_free_with_unknown_kind
/test/pmem_cpp_allocator
/test/schedcpu_test
/test/tieddisterr_test
/test/alloc_benchmark_glibc
/test/alloc_benchmark_hbw
/test/alloc_benchmark_tbb
/test/alloc_benchmark_pmem
/test/autohbw_test_helper
/test/environ_err_hbw_malloc_test
/test/trace_mechanism_test_helper
/examples/autohbw_candidates
/examples/eratosthenes
/examples/filter_memkind
/examples/hello_hbw
/examples/hello_memkind
/examples/hello_memkind_debug
/examples/memkind_allocated
/examples/new_kind
/examples/numakind_test
/examples/numakind_macro.h
/examples/pmem_kinds
/examples/pmem_malloc
/examples/pmem_malloc_unlimited
/examples/pmem_usable_size
/examples/pmem_alignment
/examples/pmem_and_default_kind
/examples/pmem_multithreads
/examples/pmem_multithreads_onekind
/examples/pmem_free_with_unknown_kind
/examples/pmem_cpp_allocator
/test-driver
/vs
+80
View File
@@ -0,0 +1,80 @@
dist: trusty
sudo: required
language: cpp
cache: ccache
matrix:
fast_finish: true
include:
- name: "Astyle"
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- g++-4.8
before_install:
- ./install_astyle.sh
script:
- ./astyle.sh
- name: "GCC 4.8"
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- g++-4.8
- libnuma-dev
env:
- MATRIX_EVAL="CC=gcc-4.8 && CXX=g++-4.8 && JEMALLOC_CC=gcc-4.8 && JEMALLOC_CXX=g++-4.8"
install:
script:
- CXX=$JEMALLOC_CXX CC=$JEMALLOC_CC LDFLAGS="-lrt -Wl,--no-as-needed" ./build_jemalloc.sh
- ./autogen.sh
- LDFLAGS="-lrt -Wl,--no-as-needed" ./configure
- make all
- make checkprogs
before_install:
- eval "${MATRIX_EVAL}"
- name: "GCC 8.0"
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- g++-8
- libnuma-dev
env:
- MATRIX_EVAL="CC=gcc-8 && CXX=g++-8 && JEMALLOC_CC=gcc-8 && JEMALLOC_CXX=g++-8"
install:
script:
- CXX=$JEMALLOC_CXX CC=$JEMALLOC_CC LDFLAGS="-lrt -Wl,--no-as-needed" ./build_jemalloc.sh
- ./autogen.sh
- LDFLAGS="-lrt -Wl,--no-as-needed" ./configure
- make all
- make checkprogs
before_install:
- eval "${MATRIX_EVAL}"
- name: "Clang 7"
addons:
apt:
sources:
- ubuntu-toolchain-r-test
- llvm-toolchain-trusty-7
packages:
- g++-4.8
- clang-7
- libnuma-dev
env:
- MATRIX_EVAL="CC=clang-7 && CXX=clang++-7 && JEMALLOC_CC=gcc-4.8 && JEMALLOC_CXX=g++-4.8"
install:
script:
- CXX=$JEMALLOC_CXX CC=$JEMALLOC_CC LDFLAGS="-lrt -Wl,--no-as-needed" ./build_jemalloc.sh
- ./autogen.sh
- LDFLAGS="-lrt -Wl,--no-as-needed" ./configure
- make all
before_install:
- eval "${MATRIX_EVAL}"
+23
View File
@@ -0,0 +1,23 @@
Christopher Cantalupo <christopher.m.cantalupo@intel.com>
Vishwanath Venkatesan <vishwanath.venkatesan@intel.com>
Steven Hampson <steven.t.hampson@intel.com>
Krzysztof Czurylo <krzysztof.czurylo@intel.com>
Leobardo Rountree <leobardo.r.rountree@intel.com>
Hector Barajas Villalobos <hector.a.barajas.villalobos@intel.com>
Ruchira Sasanka <ruchira.sasanka@intel.com>
Krzysztof Kulakowski <krzysztof.kulakowski@intel.com>
Rafael Aquini <aquini@redhat.com>
Alexandr Konovalov <alexandr.konovalov@intel.com>
Grzegorz Ozanski <grzegorz.ozanski@intel.com>
Artur Koziej <artur.koziej@intel.com>
Jakub Dlugolecki <jakub.dlugolecki@intel.com>
Lukasz Anaczkowski <lukasz.anaczkowski@intel.com>
Agata Wozniak <agata.wozniak@intel.com>
Grzegorz Andrejczuk <grzegorz.andrejczuk@intel.com>
Kamil Patelczyk <kamil.patelczyk@intel.com>
Jeff Hammond <jeff.r.hammond@intel.com>
Aleksandra Hernik <aleksandra.hernik@intel.com>
Pawel Kochanek <pawel.b.kochanek@intel.com>
Katarzyna Wasiuta <katarzyna.wasiuta@intel.com>
Krzysztof Filipek <krzysztof.filipek@intel.com>
Michal Biesek <michal.biesek@intel.com>
+134
View File
@@ -0,0 +1,134 @@
CONTRIBUTING
============
This file is intended to help those interested in contributing to the
memkind library.
COMMUNICATION
=============
Please participate in the memkind mailing list:
https://lists.01.org/mailman/listinfo/memkind
There is also the option of opening an issue through github:
https://github.com/memkind/memkind/issues
The mailing list is intended for discussion and the issues are useful
for tracking progress on tasks. The TODO file lists out a number of
topics, and in order to prioritize one of them please open a github
issue with a related subject.
TESTING
=======
The tests require a Linux kernel newer than 3.11 (the details are
documented in the memkind README), and the reservation of 3000 huge
pages. The huge pages can be reserved with the following command:
$ sudo echo 3000 > /proc/sys/vm/nr_hugepages
Only in the case where gigabyte pages have been reserved will the
tests associated with gigabyte pages be executed. Reserving gigabyte
pages may require a modification to the kernel command line unless the
kernel is quite recent.
To test memkind simply execute the "make check" target after building.
This target calls memkind/test/test.sh with parameters
depending on the environment.
Most of the tests are written within the gtest framework, however, as
part of testing the example programs are also executed and the return
code of the executable determines pass or fail. The autotools test
infrastructure is used as a high level executor, but it does not track
individual tests. The consequence of this is that the autotools
output records only one high level test which passes in the case where
every underlying test was successful and fails if any underlying test
fails. The individual test results are recorded in the directory
called "gtest_output." Here you will find the log of the tests in
gtest_output/test.out and a set of junit style xml results: one for
each test. Note that a side effect of having only one autotools test
is that autotools parallel testing is disabled. We have
multi-threaded tests that use the OpenMP run-time which enables more
purposeful and deterministic testing of threading issues. Note that
the OpenMP run-time is only required for testing, it is not used by
the memkind library internally.
CODING STYLE
============
Before submitting a patch for inclusion, please run the modified
source code files through astyle with the following options
$ astyle --style=linux --indent=spaces=4 -S --max-continuation-indent=80 \
--max-code-length=80 --break-after-logical --indent-namespaces -z2 \
--align-pointer=name
More information about astyle can be found here:
http://astyle.sourceforge.net/
In C, source code constants are in all caps and everything else is in
all lower case. Underscores are used to separate words within a
symbol name. No camel case shall be used in C code. The test code is
largely written in C++. Here camel-case should be used for class
names and should always have a capitalized first letter. Other
symbols in C++ should generally follow the C style.
Most symbols with global scope shall be prefixed with "memkind_" or
"hbw_" depending on which interface they are a part of.
Any global variable shall have _g appended to the variable name and in
most cases shall be statically scoped within a single compilation
unit. The exception to that rule are static memory kinds that are
declared as extern within the associated interface header and are
constant after initialization. Global variables should be used in a
very narrow set of circumstances and in most cases modifications
should be guarded with pthread_once(3).
Functions not used outside of the compilation unit shall be declared
as static. All functions which are not declared as static shall be
documented in a man page and have an associated interface header file.
Preprocessor mark-up is discouraged when other options are possible.
Please use enum in place of #define when value control at configure or
build time is not required.
TESTS
=====
The current state of the tests is not nearly as well organized as it
could be. That being said, it is quite simple to add a new test.
Most C++ files in the test directory are associated with a single
gtest testing::Test class. These classes usually have several
associated test fixtures in the same file. If a new test can be added
as a fixture to an existing class, simply add the fixture to the file
and the test will be incorporated into the test infrastructure.
If a new class is required, create a new file and add it to the list
of "test_all_tests_SOURCES" in memkind/test/Makfile.mk and it will
be incorporated into the test infrastructure.
There are a few files which define classes which are not google test
classes. These are check.cpp, trial_generator.cpp and main.cpp. The
check.cpp file defines a class Check that can be used to validate
fundamental memkind features like NUMA node location, and page size.
The trial_generator.cpp file is used to abstract a sequence of
allocation and deallocation calls while performing checks on the
results of each call; this can be used to apply similar tests to all
of the different allocation APIs. The main.cpp file is a simple
wrapper around testing::InitGoogleTest and RUN_ALL_TESTS().
SUBMITTING A PATCH
==================
Please be sure that all tests pass before submission and that the
style conforms to the specifications given here. If a new feature is
implemented in the patch, please also include unit tests and an
example which exercises this feature. Once these requirements have
been met, please submit a pull request through github.
+25
View File
@@ -0,0 +1,25 @@
Unless otherwise specified, files in the memkind source distribution are
subject to the following license:
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of Intel Corporation nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+219
View File
@@ -0,0 +1,219 @@
* Mon Dec 10 2018 Michal Biesek <michal.biesek@intel.com> v1.8.0
- Fixed error with realloc/free method with passing thread-specific cache flag
- Fixed error with memkind_create_pmem(), if other PMEM kind was destroyed before
- Fixed error with zeroing large allocations in PMEM kind
- Added support to create kind without maximum size limit of PMEM kind (max_size=0)
- Extended memkind API with memkind_malloc_usable_size()
- Removed EXPERIMENTAL from most methods in memkind API
- Added MEMKIND_ERROR_ARENAS_CREATE code
- Added C++ allocator for PMEM kind
- Extended PMEM examples
- Fixed integration with Travis CI
- Extended Travis CI configuration with Astyle and Coverage
- Added PMEM kind tests
* Wed Jan 24 2018 Pawel Kochanek <pawel.b.kochanek@intel.com> v1.7.0
- Updated internally used jemalloc to upstream version 5.0.
- Fixed error that has been occuring while memkind was dynamically loaded.
- Fixed MEMKIND_HBW_NODES behavior for single NUMA node system.
- Removed licenses other than BSD 3-clause from COPYING.
- Changed build instructions.
- Added configurable jemalloc prefix in build scripts.
- Upgraded gtest to version 1.8.0.
- Added memory footprint tests.
- Added locality test for MEMKIND_PREFERRED.
- Applied test parametrization in BATests.
- Fixed problems with pytest testing in Travis.
- Added huge page configuration in several tests.
- Removed several symbols that are no longer exposed in public API from man pages.
- Fixed HBW_POLICY_BIND_ALL documentation.
* Tue Jun 13 2017 Artur Koziej <artur.koziej@intel.com> v1.6.0
- Introduced new policy HBW_POLICY_BIND_ALL.
- Introduced MEMKIND_HBW_ALL, MEMKIND_HBW_ALL_HUGETLB and MEMKIND_REGULAR kinds.
- Fixed hbw_posix_memalign_psize() return value in case of unsupported policy and
page size combination.
- Documented supported policy and page size combination for
hbw_posix_memalign_psize().
- Merged libmemkind.a and libjemalloc_pic.a into a single static library.
- Fixed OOM during tcache acquisition.
- Tuned huge pages configuration in memkind tests.
- Added stress tests for allocations with large random sizes.
- Added stress tests for allocations with random kinds.
- Extended policy and NUMA node binding tests.
- Fixed several Travis build problems.
- Removed gb_page_tests_preferred_policy tests.
* Tue Feb 21 2017 Krzysztof Kulakowski <krzysztof.kulakowski@intel.com> v1.5.0
- Intel(R) TBB scalable_allocator can be used as heap management alternative
to jemalloc (requires installed Intel (R) TBB package and environment variable
MEMKIND_HEAP_MANAGER=TBB)
- Updated internally used jemalloc to upstream version 4.3.1 (previously modified 3.5.1)
- Introduced new environment option MEMKIND_HOG_MEMORY. Setting it to "1" will prevent
memkind from releasing memory to OS.
- Introduced support for KNM processor from Intel(R) Xeon Phi(TM) family
- General cleanup of build scripts (e.g. fixed issues found on Ubuntu,
introduced simple build.sh script)
- Deprecated support for gigabyte-pages
- Removed number of previously deprecated symbols
* Thu Nov 17 2016 Krzysztof Kulakowski <krzysztof.kulakowski@intel.com> v1.4.0
- Introduced hbw_verify_memory_region() for verifying if provided memory
region fully falls into high-bandwidth memory (for details please take a look
at man/hbwmalloc.3).
- New API for creating kinds using set of predefined attributes:
memkind_create_kind(), memkind_destroy_kind() (for details please take a look
at man/memkind.3).
- Fixed issue in autohbw where it was reporting limit error regardless of
value provided.
- Added logging on error paths.
- Test base fixes and improvements.
* Tue Sep 27 2016 Krzysztof Kulakowski <krzysztof.kulakowski@intel.com> v1.3.0
- Introduced logging mechanism (for details please take a look at
man/memkind.3).
- Deprecated symbols: memkind_finalize(), memkind_get_num_kind(),
memkind_get_kind_by_partition(), memkind_get_kind_by_name(),
memkind_partition_mmap(), memkind_get_size(), MEMKIND_ERROR_MEMALIGN,
MEMKIND_ERROR_MALLCTL, MEMKIND_ERROR_GETCPU, MEMKIND_ERROR_PMTT,
MEMKIND_ERROR_TIEDISTANCE, MEMKIND_ERROR_ALIGNMENT, MEMKIND_ERROR_MALLOCX,
MEMKIND_ERROR_REPNAME, MEMKIND_ERROR_PTHREAD, MEMKIND_ERROR_BADPOLICY,
MEMKIND_ERROR_REPPOLICY.
- Added integration with hwloc (turned on with --with-hwloc).
- Cleanup of symbols exposed by libmemkind.so (e.g. no longer exposing libnuma
and jemalloc symbols).
- AutoHBW files have been moved to "memkind/autohbw" directory, code has
been refactored and tests have been added to appropriate scenarios.
- Library is now built with flags improving security (can be turned off
with --disable-secure configure time option).
- Changed configuration of jemalloc to turn off unused features.
* Thu Aug 18 2016 Krzysztof Kulakowski <krzysztof.kulakowski@intel.com> v1.2.0
- memkind_create() and memkind_ops have been deprecated (moved to
memkind_deprecated.h).
- Deprecated the headers from memkind/internal and added compile-time
warnings to them (plan is to remove those from rpm packages).
- autohbw files have been moved to memkind/autohbw directory.
- API decorators (memkind_malloc_pre, memkind_malloc_post etc.) now
need to be enabled with configure-time option --enable-decorators.
- Allocation time optimizations with up to 20% improvement.
* Wed Jul 13 2016 Krzysztof Kulakowski <krzysztof.kulakowski@intel.com> v1.1.1
- Performance improvement for memkind_free() in scenario where NULL
was passed as kind (reduced by 60%).
- Introduced integration with Travis CI.
- Fixed issue where memory returned from calloc was not filled with zeroes
when using memkind_pmem kinds.
- Fixed issue where interleave kinds was failing on systems without
Transparent Huge Pages module configured.
- Resolved several issues that was causing compilation errors on some systems.
- test/test.sh is now able to detect system configuration, and run only those
tests which requirements are meet.
- Added gtest to repo to avoid downloading it during build proecess.
* Mon May 23 2016 Krzysztof Kulakowski <krzysztof.kulakowski@intel.com> v1.1.0
- Implemented algorithm for detecting high-bandwidth memory on Intel Xeon Phi X200
without parsing PMTT.
- PMTT-parsing service has been removed.
- memkind struct definition from memkind.h has been replaced by forward
declaration. Definition has been moved to /internal/memkind_private.h and
shouldn't be considered as part of library interface anymore.
- Performance improvment for hugetlb kinds (check_available() is no longer
parsing sysfs every time).
- Performance fix by removing --enable-safe from jemmaloc build flags.
- New tests for validating high-bandwidth memory detection has been added.
- AFTS has been split into two groups: memkind-afts.ts and memkind-afts-ext.ts,
based on amount of required memory (for details view test/README).
* Thu Mar 03 2016 Artur Koziej <artur.koziej@intel.com> v1.0.0
- hbwmalloc.h released as stable API.
- Introduce API standards: STANDARD API, NON-STANDARD API, EXPERIMENTAL API.
- Introduce memkind versioning API.
- Fix type names in hbwmalloc API.
- Change error codes to POSIX standard ones in hbwmalloc API.
- Move API headers to include directory.
- Move NON-STANDARD API, or EXPERIMENTAL API to include/memkind/internal directory.
- New man page (memkind_interleave.3) describing interleave kind.
- Significant documentation improvement.
- Performance fix for jemalloc - significantly decrease of library initialization time.
- Enforce 2MB alignment in jemalloc due to the Linux kernel bug (munmap() fails for
huge pages, when the size is not aligned).
- Move to systemd defined service (memkind.service), drop init.d script.
- New rpm layout (memkind and memkind-devel).
- Remove false dependency on OpenMP.
- Extend test base for stress, longevity and initialization performance tests.
- Fix memory leak in ctl_growk().
- Rename namespace: hbwmalloc -> hbw.
- Rename class: hbwmalloc_allocator -> allocator.
* Thu Oct 15 2015 Krzysztof Kulakowski <krzysztof.kulakowski@intel.com> v0.3.0
- Added two new kinds: MEMKIND_INTERLEAVE and MEMKIND_HBW_INTERLEAVE
(for details please take a look at man/memkind.3)
- Added support for file-backed memory heaps
(for details please take a look at man/memkind_pmem.3)
- Added autohbw library, that intercepts the standard heap allocations,
which let use high-bandwidth memory without modifying existing codebase
(for details please take a look at examples/autohbw_README)
- jemalloc is now, staticly linked, internal component of memkind RPM
instead of rpm dependency.
- Added memkind_allocated example which demonstrates usage of memkind
with alignas() alignment specifier introduced in C++11
(for details please take a look at examples/README)
- Added support for using thread local storage to improve performance in
multithreaded applications (enabled by configure time option: --enable-tls)
- Added posibility to set number of jemalloc arenas per kind by
configure time parameter or enviromental variable
(for details please take a look at MEMKIND_ARENA_NUM_PER_KIND section
of man/memkind.3).
- Added decorators to the memkind allocation APIs.
These are weak symbols (pre and post for each API) which can modify
the input and output of each of the calls.
- Significantly extended test base (new groups of tests: performance,
multithreaded, PMTT, nagtive, stress).
* Fri Jan 9 2015 Christopher Cantalupo <christopher.m.cantalupo@intel.com> v0.2.0
- Bumped memkind ABI version to 0:1:0.
- Removed memkind_get_kind_for_free() from externally facing API's. Instead if
memkind_free() is called with zero passed for the kind then
memkind_get_kind_for_free() is called internally.
- Moved to single callback memkind_partition_mmap() to simplify
jemalloc modifications.
- Added hooks for setting file descriptor and offset for mmap enabling
file-backed memory kinds.
- Added a void pointer called "priv" to memkind structure for storing data for
user-defined kinds.
- Removed call to sched_getcpu(), now thread id is hashed to determine the arena
index.
- Added weak symbol hooks for decorating the heap management functions.
- Fixed several issues with init.d/memkind script and spec file scriptlets that
are exposed by SLES-12.
- Introduced an example library called numakind.
The numakind library will allocate from the closest NUMA node to a thread as
measured when that thread makes its first allocation call.
- Fixed error handling in memkind_gbtlb_mmap() that could cause a segfault when
gigabyte pages are not available.
- Added tests for PMTT parser.
- Removed binary mock PMTT table from source code, replaced it with a hexdump.
- Fixed a number of issues in test scripts which were suppressing errors.
- Removed unnecessary includes from header files.
- Better error checking in example code.
- Documentation update and clean up.
* Thu Nov 13 2014 Christopher Cantalupo <christopher.m.cantalupo@intel.com> v0.1.0
- Increased test code coverage significantly.
- Fixed bug in memkind_error_message() for MEMKIND_ERROR_TOOMANY.
- Removed memkind_arena_free() API since it was redundant with memkind_default_free().
- Static memkind structs are now declared as extern in the headers and defined in the
source files files rather than being statically defined in the headers.
* Thu Oct 30 2014 Christopher Cantalupo <christopher.m.cantalupo@intel.com> v0.0.9
- Now building with autotools.
- Updated documentation.
- Fixed typo in copyright.
- Fixed test scripts to properly handle return code of each test.
- Added C++03 standard allocator that uses hbw_malloc and hbw_free.
* Tue Sep 30 2014 Christopher Cantalupo <christopher.m.cantalupo@intel.com> v0.0.8
- Added GBTLB functionality, code clean up, documentation updates,
examples directory. Examples includes stream modified to use
memkind interface. Code coverage still lacking, and documentation
incomplete.
+370
View File
@@ -0,0 +1,370 @@
Installation Instructions
*************************
Copyright (C) 1994-1996, 1999-2002, 2004-2013 Free Software Foundation,
Inc.
Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved. This file is offered as-is,
without warranty of any kind.
Basic Installation
==================
Briefly, the shell commands `./configure; make; make install' should
configure, build, and install this package. The following
more-detailed instructions are generic; see the `README' file for
instructions specific to this package. Some packages provide this
`INSTALL' file but do not implement all of the features documented
below. The lack of an optional feature in a given package is not
necessarily a bug. More recommendations for GNU packages can be found
in *note Makefile Conventions: (standards)Makefile Conventions.
The `configure' shell script attempts to guess correct values for
various system-dependent variables used during compilation. It uses
those values to create a `Makefile' in each directory of the package.
It may also create one or more `.h' files containing system-dependent
definitions. Finally, it creates a shell script `config.status' that
you can run in the future to recreate the current configuration, and a
file `config.log' containing compiler output (useful mainly for
debugging `configure').
It can also use an optional file (typically called `config.cache'
and enabled with `--cache-file=config.cache' or simply `-C') that saves
the results of its tests to speed up reconfiguring. Caching is
disabled by default to prevent problems with accidental use of stale
cache files.
If you need to do unusual things to compile the package, please try
to figure out how `configure' could check whether to do them, and mail
diffs or instructions to the address given in the `README' so they can
be considered for the next release. If you are using the cache, and at
some point `config.cache' contains results you don't want to keep, you
may remove or edit it.
The file `configure.ac' (or `configure.in') is used to create
`configure' by a program called `autoconf'. You need `configure.ac' if
you want to change it or regenerate `configure' using a newer version
of `autoconf'.
The simplest way to compile this package is:
1. `cd' to the directory containing the package's source code and type
`./configure' to configure the package for your system.
Running `configure' might take a while. While running, it prints
some messages telling which features it is checking for.
2. Type `make' to compile the package.
3. Optionally, type `make check' to run any self-tests that come with
the package, generally using the just-built uninstalled binaries.
4. Type `make install' to install the programs and any data files and
documentation. When installing into a prefix owned by root, it is
recommended that the package be configured and built as a regular
user, and only the `make install' phase executed with root
privileges.
5. Optionally, type `make installcheck' to repeat any self-tests, but
this time using the binaries in their final installed location.
This target does not install anything. Running this target as a
regular user, particularly if the prior `make install' required
root privileges, verifies that the installation completed
correctly.
6. You can remove the program binaries and object files from the
source code directory by typing `make clean'. To also remove the
files that `configure' created (so you can compile the package for
a different kind of computer), type `make distclean'. There is
also a `make maintainer-clean' target, but that is intended mainly
for the package's developers. If you use it, you may have to get
all sorts of other programs in order to regenerate files that came
with the distribution.
7. Often, you can also type `make uninstall' to remove the installed
files again. In practice, not all packages have tested that
uninstallation works correctly, even though it is required by the
GNU Coding Standards.
8. Some packages, particularly those that use Automake, provide `make
distcheck', which can by used by developers to test that all other
targets like `make install' and `make uninstall' work correctly.
This target is generally not run by end users.
Compilers and Options
=====================
Some systems require unusual options for compilation or linking that
the `configure' script does not know about. Run `./configure --help'
for details on some of the pertinent environment variables.
You can give `configure' initial values for configuration parameters
by setting variables in the command line or in the environment. Here
is an example:
./configure CC=c99 CFLAGS=-g LIBS=-lposix
*Note Defining Variables::, for more details.
Compiling For Multiple Architectures
====================================
You can compile the package for more than one kind of computer at the
same time, by placing the object files for each architecture in their
own directory. To do this, you can use GNU `make'. `cd' to the
directory where you want the object files and executables to go and run
the `configure' script. `configure' automatically checks for the
source code in the directory that `configure' is in and in `..'. This
is known as a "VPATH" build.
With a non-GNU `make', it is safer to compile the package for one
architecture at a time in the source code directory. After you have
installed the package for one architecture, use `make distclean' before
reconfiguring for another architecture.
On MacOS X 10.5 and later systems, you can create libraries and
executables that work on multiple system types--known as "fat" or
"universal" binaries--by specifying multiple `-arch' options to the
compiler but only a single `-arch' option to the preprocessor. Like
this:
./configure CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
CPP="gcc -E" CXXCPP="g++ -E"
This is not guaranteed to produce working output in all cases, you
may have to build one architecture at a time and combine the results
using the `lipo' tool if you have problems.
Installation Names
==================
By default, `make install' installs the package's commands under
`/usr/local/bin', include files under `/usr/local/include', etc. You
can specify an installation prefix other than `/usr/local' by giving
`configure' the option `--prefix=PREFIX', where PREFIX must be an
absolute file name.
You can specify separate installation prefixes for
architecture-specific files and architecture-independent files. If you
pass the option `--exec-prefix=PREFIX' to `configure', the package uses
PREFIX as the prefix for installing programs and libraries.
Documentation and other data files still use the regular prefix.
In addition, if you use an unusual directory layout you can give
options like `--bindir=DIR' to specify different values for particular
kinds of files. Run `configure --help' for a list of the directories
you can set and what kinds of files go in them. In general, the
default for these options is expressed in terms of `${prefix}', so that
specifying just `--prefix' will affect all of the other directory
specifications that were not explicitly provided.
The most portable way to affect installation locations is to pass the
correct locations to `configure'; however, many packages provide one or
both of the following shortcuts of passing variable assignments to the
`make install' command line to change installation locations without
having to reconfigure or recompile.
The first method involves providing an override variable for each
affected directory. For example, `make install
prefix=/alternate/directory' will choose an alternate location for all
directory configuration variables that were expressed in terms of
`${prefix}'. Any directories that were specified during `configure',
but not in terms of `${prefix}', must each be overridden at install
time for the entire installation to be relocated. The approach of
makefile variable overrides for each directory variable is required by
the GNU Coding Standards, and ideally causes no recompilation.
However, some platforms have known limitations with the semantics of
shared libraries that end up requiring recompilation when using this
method, particularly noticeable in packages that use GNU Libtool.
The second method involves providing the `DESTDIR' variable. For
example, `make install DESTDIR=/alternate/directory' will prepend
`/alternate/directory' before all installation names. The approach of
`DESTDIR' overrides is not required by the GNU Coding Standards, and
does not work on platforms that have drive letters. On the other hand,
it does better at avoiding recompilation issues, and works well even
when some directory options were not specified in terms of `${prefix}'
at `configure' time.
Optional Features
=================
If the package supports it, you can cause programs to be installed
with an extra prefix or suffix on their names by giving `configure' the
option `--program-prefix=PREFIX' or `--program-suffix=SUFFIX'.
Some packages pay attention to `--enable-FEATURE' options to
`configure', where FEATURE indicates an optional part of the package.
They may also pay attention to `--with-PACKAGE' options, where PACKAGE
is something like `gnu-as' or `x' (for the X Window System). The
`README' should mention any `--enable-' and `--with-' options that the
package recognizes.
For packages that use the X Window System, `configure' can usually
find the X include and library files automatically, but if it doesn't,
you can use the `configure' options `--x-includes=DIR' and
`--x-libraries=DIR' to specify their locations.
Some packages offer the ability to configure how verbose the
execution of `make' will be. For these packages, running `./configure
--enable-silent-rules' sets the default to minimal output, which can be
overridden with `make V=1'; while running `./configure
--disable-silent-rules' sets the default to verbose, which can be
overridden with `make V=0'.
Particular systems
==================
On HP-UX, the default C compiler is not ANSI C compatible. If GNU
CC is not installed, it is recommended to use the following options in
order to use an ANSI C compiler:
./configure CC="cc -Ae -D_XOPEN_SOURCE=500"
and if that doesn't work, install pre-built binaries of GCC for HP-UX.
HP-UX `make' updates targets which have the same time stamps as
their prerequisites, which makes it generally unusable when shipped
generated files such as `configure' are involved. Use GNU `make'
instead.
On OSF/1 a.k.a. Tru64, some versions of the default C compiler cannot
parse its `<wchar.h>' header file. The option `-nodtk' can be used as
a workaround. If GNU CC is not installed, it is therefore recommended
to try
./configure CC="cc"
and if that doesn't work, try
./configure CC="cc -nodtk"
On Solaris, don't put `/usr/ucb' early in your `PATH'. This
directory contains several dysfunctional programs; working variants of
these programs are available in `/usr/bin'. So, if you need `/usr/ucb'
in your `PATH', put it _after_ `/usr/bin'.
On Haiku, software installed for all users goes in `/boot/common',
not `/usr/local'. It is recommended to use the following options:
./configure --prefix=/boot/common
Specifying the System Type
==========================
There may be some features `configure' cannot figure out
automatically, but needs to determine by the type of machine the package
will run on. Usually, assuming the package is built to be run on the
_same_ architectures, `configure' can figure that out, but if it prints
a message saying it cannot guess the machine type, give it the
`--build=TYPE' option. TYPE can either be a short name for the system
type, such as `sun4', or a canonical name which has the form:
CPU-COMPANY-SYSTEM
where SYSTEM can have one of these forms:
OS
KERNEL-OS
See the file `config.sub' for the possible values of each field. If
`config.sub' isn't included in this package, then this package doesn't
need to know the machine type.
If you are _building_ compiler tools for cross-compiling, you should
use the option `--target=TYPE' to select the type of system they will
produce code for.
If you want to _use_ a cross compiler, that generates code for a
platform different from the build platform, you should specify the
"host" platform (i.e., that on which the generated programs will
eventually be run) with `--host=TYPE'.
Sharing Defaults
================
If you want to set default values for `configure' scripts to share,
you can create a site shell script called `config.site' that gives
default values for variables like `CC', `cache_file', and `prefix'.
`configure' looks for `PREFIX/share/config.site' if it exists, then
`PREFIX/etc/config.site' if it exists. Or, you can set the
`CONFIG_SITE' environment variable to the location of the site script.
A warning: not all `configure' scripts look for a site script.
Defining Variables
==================
Variables not defined in a site shell script can be set in the
environment passed to `configure'. However, some packages may run
configure again during the build, and the customized values of these
variables may be lost. In order to avoid this problem, you should set
them in the `configure' command line, using `VAR=value'. For example:
./configure CC=/usr/local2/bin/gcc
causes the specified `gcc' to be used as the C compiler (unless it is
overridden in the site shell script).
Unfortunately, this technique does not work for `CONFIG_SHELL' due to
an Autoconf limitation. Until the limitation is lifted, you can use
this workaround:
CONFIG_SHELL=/bin/bash ./configure CONFIG_SHELL=/bin/bash
`configure' Invocation
======================
`configure' recognizes the following options to control how it
operates.
`--help'
`-h'
Print a summary of all of the options to `configure', and exit.
`--help=short'
`--help=recursive'
Print a summary of the options unique to this package's
`configure', and exit. The `short' variant lists options used
only in the top level, while the `recursive' variant lists options
also present in any nested packages.
`--version'
`-V'
Print the version of Autoconf used to generate the `configure'
script, and exit.
`--cache-file=FILE'
Enable the cache: use and save the results of the tests in FILE,
traditionally `config.cache'. FILE defaults to `/dev/null' to
disable caching.
`--config-cache'
`-C'
Alias for `--cache-file=config.cache'.
`--quiet'
`--silent'
`-q'
Do not print messages saying which checks are being made. To
suppress all normal output, redirect it to `/dev/null' (any error
messages will still be shown).
`--srcdir=DIR'
Look for the package's source code in directory DIR. Usually
`configure' can determine that directory automatically.
`--prefix=DIR'
Use DIR as the installation prefix. *note Installation Names::
for more details, including other options available for fine-tuning
the installation locations.
`--no-create'
`-n'
Run the configure checks, but stop before creating any output
files.
`configure' also accepts some other, not widely useful, options. Run
`configure --help' for more details.
+469
View File
@@ -0,0 +1,469 @@
AUTHORS
COPYING
CONTRIBUTING
ChangeLog
INSTALL
Makefile.am
NEWS
PULL_REQUEST_TEMPLATE.md
README
.travis.yml
astyle.sh
autogen.sh
build.sh
build_jemalloc.sh
configure.ac
autohbw/Makefile.mk
autohbw/autohbw.c
autohbw/autohbw_get_src_lines.pl
autohbw/autohbw_README
autohbw/autohbw_test.sh
examples/autohbw_candidates.c
examples/filter_example.c
examples/hello_memkind_example.c
examples/README
examples/Makefile.mk
examples/hello_hbw_example.c
examples/memkind_allocated_example.cpp
examples/memkind_allocated.hpp
examples/memkind_decorator_debug.c
examples/pmem_kinds.c
examples/pmem_malloc.c
examples/pmem_malloc_unlimited.c
examples/pmem_usable_size.c
examples/pmem_alignment.c
examples/pmem_and_default_kind.c
examples/pmem_multithreads.c
examples/pmem_multithreads_onekind.c
examples/pmem_free_with_unknown_kind.c
examples/pmem_cpp_allocator.cpp
install_astyle.sh
m4/ax_cxx_compile_stdcxx.m4
m4/ax_cxx_compile_stdcxx_11.m4
m4/ax_pthread.m4
man/memkind-hbw-nodes.1
man/hbwmalloc.3
man/memkind.3
man/memkind_arena.3
man/memkind_default.3
man/memkind_hbw.3
man/memkind_hugetlb.3
man/memkind_pmem.3
man/hbwallocator.3
man/pmemallocator.3
man/autohbw.7
memkind.spec.mk
src/heap_manager.c
src/hbwmalloc.c
src/memkind.c
src/memkind_arena.c
src/memkind_default.c
src/memkind_gbtlb.c
src/memkind_hbw.c
src/memkind_regular.c
src/memkind_hugetlb.c
src/memkind_interleave.c
src/memkind_pmem.c
src/memkind_log.c
src/memkind-hbw-nodes.c
src/tbb_wrapper.c
src/Makefile.mk
include/hbwmalloc.h
include/memkind.h
include/memkind_deprecated.h
include/hbw_allocator.h
include/pmem_allocator.h
include/memkind/internal/heap_manager.h
include/memkind/internal/memkind_arena.h
include/memkind/internal/memkind_default.h
include/memkind/internal/memkind_gbtlb.h
include/memkind/internal/memkind_hbw.h
include/memkind/internal/memkind_regular.h
include/memkind/internal/memkind_hugetlb.h
include/memkind/internal/memkind_interleave.h
include/memkind/internal/memkind_pmem.h
include/memkind/internal/memkind_private.h
include/memkind/internal/memkind_log.h
include/memkind/internal/tbb_wrapper.h
include/memkind/internal/tbb_mem_pool_policy.h
test/Makefile.mk
test/Allocator.hpp
test/TestPolicy.hpp
test/test.sh
test/multithreaded_tests.cpp
test/gb_page_tests_bind_policy.cpp
test/environ_err_hbw_malloc_test.cpp
test/negative_tests.cpp
test/trial_generator.cpp
test/check.h
test/trial_generator.h
test/bat_tests.cpp
test/main.cpp
test/common.h
test/static_kinds_list.h
test/check.cpp
test/error_message_tests.cpp
test/get_arena_test.cpp
test/locality_test.cpp
test/memkind_pmem_tests.cpp
test/memkind_pmem_long_time_tests.cpp
test/memory_footprint_test.cpp
test/memory_manager.h
test/random_sizes_allocator.h
test/proc_stat.h
test/static_kinds_tests.cpp
test/huge_page_test.cpp
test/hbw_verify_function_test.cpp
test/hbw_detection_test.py
test/autohbw_test.py
test/trace_mechanism_test.py
test/memkind-afts.ts
test/memkind-afts-ext.ts
test/memkind-slts.ts
test/memkind-perf.ts
test/memkind-perf-ext.ts
test/memkind-pytests.ts
test/performance/framework.hpp
test/performance/framework.cpp
test/performance/operations.hpp
test/performance/perf_tests.hpp
test/performance/perf_tests.cpp
test/hbw_allocator_tests.cpp
test/decorator_test.cpp
test/decorator_test.h
test/alloc_performance_tests.cpp
test/allocator_perf_tool/AllocationSizes.hpp
test/allocator_perf_tool/Allocation_info.hpp
test/allocator_perf_tool/Allocation_info.cpp
test/allocator_perf_tool/Allocator.hpp
test/allocator_perf_tool/AllocatorFactory.hpp
test/allocator_perf_tool/CSVLogger.hpp
test/allocator_perf_tool/CommandLine.hpp
test/allocator_perf_tool/Configuration.hpp
test/allocator_perf_tool/ConsoleLog.hpp
test/allocator_perf_tool/FunctionCalls.hpp
test/allocator_perf_tool/FunctionCallsPerformanceTask.cpp
test/allocator_perf_tool/FunctionCallsPerformanceTask.h
test/allocator_perf_tool/GTestAdapter.hpp
test/allocator_perf_tool/Iterator.hpp
test/allocator_perf_tool/JemallocAllocatorWithTimer.hpp
test/allocator_perf_tool/Makefile
test/allocator_perf_tool/MemkindAllocatorWithTimer.hpp
test/allocator_perf_tool/Numastat.hpp
test/allocator_perf_tool/Runnable.hpp
test/allocator_perf_tool/PmemMockup.cpp
test/allocator_perf_tool/PmemMockup.hpp
test/allocator_perf_tool/ScenarioWorkload.cpp
test/allocator_perf_tool/ScenarioWorkload.h
test/allocator_perf_tool/StandardAllocatorWithTimer.hpp
test/allocator_perf_tool/Stats.hpp
test/allocator_perf_tool/StressIncreaseToMax.cpp
test/allocator_perf_tool/StressIncreaseToMax.h
test/allocator_perf_tool/Task.hpp
test/allocator_perf_tool/TaskFactory.hpp
test/allocator_perf_tool/Tests.hpp
test/allocator_perf_tool/Thread.hpp
test/allocator_perf_tool/TimerSysTime.hpp
test/allocator_perf_tool/VectorIterator.hpp
test/allocator_perf_tool/Workload.hpp
test/allocator_perf_tool/WrappersMacros.hpp
test/allocator_perf_tool/HugePageUnmap.hpp
test/allocator_perf_tool/HugePageOrganizer.hpp
test/allocator_perf_tool/HBWmallocAllocatorWithTimer.hpp
test/allocator_perf_tool/main.cpp
test/allocate_to_max_stress_test.cpp
test/memkind_versioning_tests.cpp
test/heap_manager_init_perf_test.cpp
test/autohbw_test_helper.c
test/trace_mechanism_test_helper.c
test/python_framework/cmd_helper.py
test/python_framework/huge_page_organizer.py
test/python_framework/__init__.py
test/draw_plots.py
test/run_alloc_benchmark.sh
test/alloc_benchmark.c
test/load_tbbmalloc_symbols.c
test/tbbmalloc.h
test/freeing_memory_segfault_test.cpp
test/dlopen_test.cpp
test/gtest_fused/gtest/gtest-all.cc
test/gtest_fused/gtest/gtest.h
test/pmem_allocator_tests.cpp
jemalloc/.appveyor.yml
jemalloc/.autom4te.cfg
jemalloc/.gitattributes
jemalloc/.gitignore
jemalloc/.travis.yml
jemalloc/COPYING
jemalloc/ChangeLog
jemalloc/INSTALL.md
jemalloc/Makefile.in
jemalloc/README
jemalloc/autogen.sh
jemalloc/bin/jemalloc-config.in
jemalloc/bin/jemalloc.sh.in
jemalloc/bin/jeprof.in
jemalloc/build-aux/config.guess
jemalloc/build-aux/config.sub
jemalloc/build-aux/install-sh
jemalloc/config.stamp.in
jemalloc/configure.ac
jemalloc/doc/html.xsl.in
jemalloc/doc/jemalloc.xml.in
jemalloc/doc/manpages.xsl.in
jemalloc/doc/stylesheet.xsl
jemalloc/include/jemalloc/internal/arena_externs.h
jemalloc/include/jemalloc/internal/arena_inlines_a.h
jemalloc/include/jemalloc/internal/arena_inlines_b.h
jemalloc/include/jemalloc/internal/arena_structs_a.h
jemalloc/include/jemalloc/internal/arena_structs_b.h
jemalloc/include/jemalloc/internal/arena_types.h
jemalloc/include/jemalloc/internal/assert.h
jemalloc/include/jemalloc/internal/atomic.h
jemalloc/include/jemalloc/internal/atomic_c11.h
jemalloc/include/jemalloc/internal/atomic_gcc_atomic.h
jemalloc/include/jemalloc/internal/atomic_gcc_sync.h
jemalloc/include/jemalloc/internal/atomic_msvc.h
jemalloc/include/jemalloc/internal/background_thread_externs.h
jemalloc/include/jemalloc/internal/background_thread_inlines.h
jemalloc/include/jemalloc/internal/background_thread_structs.h
jemalloc/include/jemalloc/internal/base_externs.h
jemalloc/include/jemalloc/internal/base_inlines.h
jemalloc/include/jemalloc/internal/base_structs.h
jemalloc/include/jemalloc/internal/base_types.h
jemalloc/include/jemalloc/internal/bit_util.h
jemalloc/include/jemalloc/internal/bitmap.h
jemalloc/include/jemalloc/internal/ckh.h
jemalloc/include/jemalloc/internal/ctl.h
jemalloc/include/jemalloc/internal/extent_dss.h
jemalloc/include/jemalloc/internal/extent_externs.h
jemalloc/include/jemalloc/internal/extent_inlines.h
jemalloc/include/jemalloc/internal/extent_mmap.h
jemalloc/include/jemalloc/internal/extent_structs.h
jemalloc/include/jemalloc/internal/extent_types.h
jemalloc/include/jemalloc/internal/hash.h
jemalloc/include/jemalloc/internal/hooks.h
jemalloc/include/jemalloc/internal/jemalloc_internal_decls.h
jemalloc/include/jemalloc/internal/jemalloc_internal_defs.h.in
jemalloc/include/jemalloc/internal/jemalloc_internal_externs.h
jemalloc/include/jemalloc/internal/jemalloc_internal_includes.h
jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_a.h
jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_b.h
jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_c.h
jemalloc/include/jemalloc/internal/jemalloc_internal_macros.h
jemalloc/include/jemalloc/internal/jemalloc_internal_types.h
jemalloc/include/jemalloc/internal/jemalloc_preamble.h.in
jemalloc/include/jemalloc/internal/large_externs.h
jemalloc/include/jemalloc/internal/malloc_io.h
jemalloc/include/jemalloc/internal/mutex.h
jemalloc/include/jemalloc/internal/mutex_pool.h
jemalloc/include/jemalloc/internal/mutex_prof.h
jemalloc/include/jemalloc/internal/nstime.h
jemalloc/include/jemalloc/internal/pages.h
jemalloc/include/jemalloc/internal/ph.h
jemalloc/include/jemalloc/internal/private_namespace.sh
jemalloc/include/jemalloc/internal/private_symbols.sh
jemalloc/include/jemalloc/internal/prng.h
jemalloc/include/jemalloc/internal/prof_externs.h
jemalloc/include/jemalloc/internal/prof_inlines_a.h
jemalloc/include/jemalloc/internal/prof_inlines_b.h
jemalloc/include/jemalloc/internal/prof_structs.h
jemalloc/include/jemalloc/internal/prof_types.h
jemalloc/include/jemalloc/internal/public_namespace.sh
jemalloc/include/jemalloc/internal/public_unnamespace.sh
jemalloc/include/jemalloc/internal/ql.h
jemalloc/include/jemalloc/internal/qr.h
jemalloc/include/jemalloc/internal/rb.h
jemalloc/include/jemalloc/internal/rtree.h
jemalloc/include/jemalloc/internal/rtree_tsd.h
jemalloc/include/jemalloc/internal/size_classes.sh
jemalloc/include/jemalloc/internal/smoothstep.h
jemalloc/include/jemalloc/internal/smoothstep.sh
jemalloc/include/jemalloc/internal/spin.h
jemalloc/include/jemalloc/internal/stats.h
jemalloc/include/jemalloc/internal/stats_tsd.h
jemalloc/include/jemalloc/internal/sz.h
jemalloc/include/jemalloc/internal/tcache_externs.h
jemalloc/include/jemalloc/internal/tcache_inlines.h
jemalloc/include/jemalloc/internal/tcache_structs.h
jemalloc/include/jemalloc/internal/tcache_types.h
jemalloc/include/jemalloc/internal/ticker.h
jemalloc/include/jemalloc/internal/tsd.h
jemalloc/include/jemalloc/internal/tsd_generic.h
jemalloc/include/jemalloc/internal/tsd_malloc_thread_cleanup.h
jemalloc/include/jemalloc/internal/tsd_tls.h
jemalloc/include/jemalloc/internal/tsd_types.h
jemalloc/include/jemalloc/internal/tsd_win.h
jemalloc/include/jemalloc/internal/util.h
jemalloc/include/jemalloc/internal/witness.h
jemalloc/include/jemalloc/jemalloc.sh
jemalloc/include/jemalloc/jemalloc_defs.h.in
jemalloc/include/jemalloc/jemalloc_macros.h.in
jemalloc/include/jemalloc/jemalloc_mangle.sh
jemalloc/include/jemalloc/jemalloc_protos.h.in
jemalloc/include/jemalloc/jemalloc_rename.sh
jemalloc/include/jemalloc/jemalloc_typedefs.h.in
jemalloc/include/msvc_compat/C99/stdbool.h
jemalloc/include/msvc_compat/C99/stdint.h
jemalloc/include/msvc_compat/strings.h
jemalloc/include/msvc_compat/windows_extra.h
jemalloc/jemalloc.pc.in
jemalloc/m4/ax_cxx_compile_stdcxx.m4
jemalloc/msvc/ReadMe.txt
jemalloc/msvc/jemalloc_vc2015.sln
jemalloc/msvc/projects/vc2015/jemalloc/jemalloc.vcxproj
jemalloc/msvc/projects/vc2015/jemalloc/jemalloc.vcxproj.filters
jemalloc/msvc/projects/vc2015/test_threads/test_threads.cpp
jemalloc/msvc/projects/vc2015/test_threads/test_threads.h
jemalloc/msvc/projects/vc2015/test_threads/test_threads.vcxproj
jemalloc/msvc/projects/vc2015/test_threads/test_threads.vcxproj.filters
jemalloc/msvc/projects/vc2015/test_threads/test_threads_main.cpp
jemalloc/run_tests.sh
jemalloc/scripts/gen_run_tests.py
jemalloc/scripts/gen_travis.py
jemalloc/src/arena.c
jemalloc/src/background_thread.c
jemalloc/src/base.c
jemalloc/src/bitmap.c
jemalloc/src/ckh.c
jemalloc/src/ctl.c
jemalloc/src/extent.c
jemalloc/src/extent_dss.c
jemalloc/src/extent_mmap.c
jemalloc/src/hash.c
jemalloc/src/hooks.c
jemalloc/src/jemalloc.c
jemalloc/src/jemalloc_cpp.cpp
jemalloc/src/large.c
jemalloc/src/malloc_io.c
jemalloc/src/mutex.c
jemalloc/src/mutex_pool.c
jemalloc/src/nstime.c
jemalloc/src/pages.c
jemalloc/src/prng.c
jemalloc/src/prof.c
jemalloc/src/rtree.c
jemalloc/src/spin.c
jemalloc/src/stats.c
jemalloc/src/sz.c
jemalloc/src/tcache.c
jemalloc/src/ticker.c
jemalloc/src/tsd.c
jemalloc/src/witness.c
jemalloc/src/zone.c
jemalloc/test/include/test/SFMT-alti.h
jemalloc/test/include/test/SFMT-params.h
jemalloc/test/include/test/SFMT-params11213.h
jemalloc/test/include/test/SFMT-params1279.h
jemalloc/test/include/test/SFMT-params132049.h
jemalloc/test/include/test/SFMT-params19937.h
jemalloc/test/include/test/SFMT-params216091.h
jemalloc/test/include/test/SFMT-params2281.h
jemalloc/test/include/test/SFMT-params4253.h
jemalloc/test/include/test/SFMT-params44497.h
jemalloc/test/include/test/SFMT-params607.h
jemalloc/test/include/test/SFMT-params86243.h
jemalloc/test/include/test/SFMT-sse2.h
jemalloc/test/include/test/SFMT.h
jemalloc/test/include/test/btalloc.h
jemalloc/test/include/test/extent_hooks.h
jemalloc/test/include/test/jemalloc_test.h.in
jemalloc/test/include/test/jemalloc_test_defs.h.in
jemalloc/test/include/test/math.h
jemalloc/test/include/test/mq.h
jemalloc/test/include/test/mtx.h
jemalloc/test/include/test/test.h
jemalloc/test/include/test/thd.h
jemalloc/test/include/test/timer.h
jemalloc/test/integration/MALLOCX_ARENA.c
jemalloc/test/integration/aligned_alloc.c
jemalloc/test/integration/allocated.c
jemalloc/test/integration/extent.c
jemalloc/test/integration/extent.sh
jemalloc/test/integration/mallocx.c
jemalloc/test/integration/mallocx.sh
jemalloc/test/integration/overflow.c
jemalloc/test/integration/posix_memalign.c
jemalloc/test/integration/rallocx.c
jemalloc/test/integration/sdallocx.c
jemalloc/test/integration/thread_arena.c
jemalloc/test/integration/thread_tcache_enabled.c
jemalloc/test/integration/xallocx.c
jemalloc/test/integration/xallocx.sh
jemalloc/test/src/SFMT.c
jemalloc/test/src/btalloc.c
jemalloc/test/src/btalloc_0.c
jemalloc/test/src/btalloc_1.c
jemalloc/test/src/math.c
jemalloc/test/src/mq.c
jemalloc/test/src/mtx.c
jemalloc/test/src/test.c
jemalloc/test/src/thd.c
jemalloc/test/src/timer.c
jemalloc/test/stress/microbench.c
jemalloc/test/test.sh.in
jemalloc/test/unit/SFMT.c
jemalloc/test/unit/a0.c
jemalloc/test/unit/arena_reset.c
jemalloc/test/unit/arena_reset_prof.c
jemalloc/test/unit/arena_reset_prof.sh
jemalloc/test/unit/atomic.c
jemalloc/test/unit/background_thread.c
jemalloc/test/unit/base.c
jemalloc/test/unit/bit_util.c
jemalloc/test/unit/bitmap.c
jemalloc/test/unit/ckh.c
jemalloc/test/unit/decay.c
jemalloc/test/unit/decay.sh
jemalloc/test/unit/extent_quantize.c
jemalloc/test/unit/fork.c
jemalloc/test/unit/hash.c
jemalloc/test/unit/hooks.c
jemalloc/test/unit/junk.c
jemalloc/test/unit/junk.sh
jemalloc/test/unit/junk_alloc.c
jemalloc/test/unit/junk_alloc.sh
jemalloc/test/unit/junk_free.c
jemalloc/test/unit/junk_free.sh
jemalloc/test/unit/mallctl.c
jemalloc/test/unit/malloc_io.c
jemalloc/test/unit/math.c
jemalloc/test/unit/mq.c
jemalloc/test/unit/mtx.c
jemalloc/test/unit/nstime.c
jemalloc/test/unit/pack.c
jemalloc/test/unit/pack.sh
jemalloc/test/unit/pages.c
jemalloc/test/unit/ph.c
jemalloc/test/unit/prng.c
jemalloc/test/unit/prof_accum.c
jemalloc/test/unit/prof_accum.sh
jemalloc/test/unit/prof_active.c
jemalloc/test/unit/prof_active.sh
jemalloc/test/unit/prof_gdump.c
jemalloc/test/unit/prof_gdump.sh
jemalloc/test/unit/prof_idump.c
jemalloc/test/unit/prof_idump.sh
jemalloc/test/unit/prof_reset.c
jemalloc/test/unit/prof_reset.sh
jemalloc/test/unit/prof_tctx.c
jemalloc/test/unit/prof_tctx.sh
jemalloc/test/unit/prof_thread_name.c
jemalloc/test/unit/prof_thread_name.sh
jemalloc/test/unit/ql.c
jemalloc/test/unit/qr.c
jemalloc/test/unit/rb.c
jemalloc/test/unit/retained.c
jemalloc/test/unit/rtree.c
jemalloc/test/unit/size_classes.c
jemalloc/test/unit/slab.c
jemalloc/test/unit/smoothstep.c
jemalloc/test/unit/spin.c
jemalloc/test/unit/stats.c
jemalloc/test/unit/stats_print.c
jemalloc/test/unit/ticker.c
jemalloc/test/unit/tsd.c
jemalloc/test/unit/witness.c
jemalloc/test/unit/zero.c
jemalloc/test/unit/zero.sh
build_jemalloc.sh
+435
View File
@@ -0,0 +1,435 @@
#
# Copyright (C) 2014 - 2018 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
ACLOCAL_AMFLAGS = -I m4
AM_CPPFLAGS = -I$(top_srcdir)/include -I$(top_srcdir)/jemalloc/obj/include
lib_LTLIBRARIES = libmemkind.la
libmemkind_la_SOURCES = src/hbwmalloc.c \
src/heap_manager.c \
src/memkind_arena.c \
src/memkind.c \
src/memkind_default.c \
src/memkind_gbtlb.c \
src/memkind_hbw.c \
src/memkind_regular.c \
src/memkind_hugetlb.c \
src/memkind_pmem.c \
src/memkind_interleave.c \
src/memkind_log.c \
src/tbb_wrapper.c \
# end
libmemkind_la_LIBADD = jemalloc/obj/lib/libjemalloc_pic.a
libmemkind_la_LDFLAGS = -version-info 0:1:0 -ldl
include_HEADERS = include/hbwmalloc.h \
include/hbw_allocator.h \
include/pmem_allocator.h \
include/memkind.h \
include/memkind_deprecated.h \
# end
nonstandardincludedir = $(includedir)/memkind/internal
nonstandardinclude_HEADERS = include/memkind/internal/memkind_arena.h \
include/memkind/internal/heap_manager.h \
include/memkind/internal/memkind_default.h \
include/memkind/internal/memkind_gbtlb.h \
include/memkind/internal/memkind_hbw.h \
include/memkind/internal/memkind_regular.h \
include/memkind/internal/memkind_hugetlb.h \
include/memkind/internal/memkind_interleave.h \
include/memkind/internal/memkind_pmem.h \
include/memkind/internal/memkind_private.h \
include/memkind/internal/memkind_log.h \
include/memkind/internal/tbb_wrapper.h \
include/memkind/internal/tbb_mem_pool_policy.h \
# end
EXTRA_DIST = astyle.sh \
autogen.sh \
build.sh \
build_jemalloc.sh \
examples/README \
test/test.sh \
VERSION \
CONTRIBUTING \
memkind-$(VERSION).spec \
memkind.spec.mk \
jemalloc/.appveyor.yml \
jemalloc/.autom4te.cfg \
jemalloc/.gitattributes \
jemalloc/.gitignore \
jemalloc/.travis.yml \
jemalloc/COPYING \
jemalloc/ChangeLog \
jemalloc/INSTALL.md \
jemalloc/Makefile.in \
jemalloc/README \
jemalloc/autogen.sh \
jemalloc/bin/jemalloc-config.in \
jemalloc/bin/jemalloc.sh.in \
jemalloc/bin/jeprof.in \
jemalloc/build-aux/config.guess \
jemalloc/build-aux/config.sub \
jemalloc/build-aux/install-sh \
jemalloc/config.stamp.in \
jemalloc/configure.ac \
jemalloc/doc/html.xsl.in \
jemalloc/doc/jemalloc.xml.in \
jemalloc/doc/manpages.xsl.in \
jemalloc/doc/stylesheet.xsl \
jemalloc/include/jemalloc/internal/arena_externs.h \
jemalloc/include/jemalloc/internal/arena_inlines_a.h \
jemalloc/include/jemalloc/internal/arena_inlines_b.h \
jemalloc/include/jemalloc/internal/arena_structs_a.h \
jemalloc/include/jemalloc/internal/arena_structs_b.h \
jemalloc/include/jemalloc/internal/arena_types.h \
jemalloc/include/jemalloc/internal/assert.h \
jemalloc/include/jemalloc/internal/atomic.h \
jemalloc/include/jemalloc/internal/atomic_c11.h \
jemalloc/include/jemalloc/internal/atomic_gcc_atomic.h \
jemalloc/include/jemalloc/internal/atomic_gcc_sync.h \
jemalloc/include/jemalloc/internal/atomic_msvc.h \
jemalloc/include/jemalloc/internal/background_thread_externs.h \
jemalloc/include/jemalloc/internal/background_thread_inlines.h \
jemalloc/include/jemalloc/internal/background_thread_structs.h \
jemalloc/include/jemalloc/internal/base_externs.h \
jemalloc/include/jemalloc/internal/base_inlines.h \
jemalloc/include/jemalloc/internal/base_structs.h \
jemalloc/include/jemalloc/internal/base_types.h \
jemalloc/include/jemalloc/internal/bit_util.h \
jemalloc/include/jemalloc/internal/bitmap.h \
jemalloc/include/jemalloc/internal/ckh.h \
jemalloc/include/jemalloc/internal/ctl.h \
jemalloc/include/jemalloc/internal/extent_dss.h \
jemalloc/include/jemalloc/internal/extent_externs.h \
jemalloc/include/jemalloc/internal/extent_inlines.h \
jemalloc/include/jemalloc/internal/extent_mmap.h \
jemalloc/include/jemalloc/internal/extent_structs.h \
jemalloc/include/jemalloc/internal/extent_types.h \
jemalloc/include/jemalloc/internal/hash.h \
jemalloc/include/jemalloc/internal/hooks.h \
jemalloc/include/jemalloc/internal/jemalloc_internal_decls.h \
jemalloc/include/jemalloc/internal/jemalloc_internal_defs.h.in \
jemalloc/include/jemalloc/internal/jemalloc_internal_externs.h \
jemalloc/include/jemalloc/internal/jemalloc_internal_includes.h \
jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_a.h \
jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_b.h \
jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_c.h \
jemalloc/include/jemalloc/internal/jemalloc_internal_macros.h \
jemalloc/include/jemalloc/internal/jemalloc_internal_types.h \
jemalloc/include/jemalloc/internal/jemalloc_preamble.h.in \
jemalloc/include/jemalloc/internal/large_externs.h \
jemalloc/include/jemalloc/internal/malloc_io.h \
jemalloc/include/jemalloc/internal/mutex.h \
jemalloc/include/jemalloc/internal/mutex_pool.h \
jemalloc/include/jemalloc/internal/mutex_prof.h \
jemalloc/include/jemalloc/internal/nstime.h \
jemalloc/include/jemalloc/internal/pages.h \
jemalloc/include/jemalloc/internal/ph.h \
jemalloc/include/jemalloc/internal/private_namespace.sh \
jemalloc/include/jemalloc/internal/private_symbols.sh \
jemalloc/include/jemalloc/internal/prng.h \
jemalloc/include/jemalloc/internal/prof_externs.h \
jemalloc/include/jemalloc/internal/prof_inlines_a.h \
jemalloc/include/jemalloc/internal/prof_inlines_b.h \
jemalloc/include/jemalloc/internal/prof_structs.h \
jemalloc/include/jemalloc/internal/prof_types.h \
jemalloc/include/jemalloc/internal/public_namespace.sh \
jemalloc/include/jemalloc/internal/public_unnamespace.sh \
jemalloc/include/jemalloc/internal/ql.h \
jemalloc/include/jemalloc/internal/qr.h \
jemalloc/include/jemalloc/internal/rb.h \
jemalloc/include/jemalloc/internal/rtree.h \
jemalloc/include/jemalloc/internal/rtree_tsd.h \
jemalloc/include/jemalloc/internal/size_classes.sh \
jemalloc/include/jemalloc/internal/smoothstep.h \
jemalloc/include/jemalloc/internal/smoothstep.sh \
jemalloc/include/jemalloc/internal/spin.h \
jemalloc/include/jemalloc/internal/stats.h \
jemalloc/include/jemalloc/internal/stats_tsd.h \
jemalloc/include/jemalloc/internal/sz.h \
jemalloc/include/jemalloc/internal/tcache_externs.h \
jemalloc/include/jemalloc/internal/tcache_inlines.h \
jemalloc/include/jemalloc/internal/tcache_structs.h \
jemalloc/include/jemalloc/internal/tcache_types.h \
jemalloc/include/jemalloc/internal/ticker.h \
jemalloc/include/jemalloc/internal/tsd.h \
jemalloc/include/jemalloc/internal/tsd_generic.h \
jemalloc/include/jemalloc/internal/tsd_malloc_thread_cleanup.h \
jemalloc/include/jemalloc/internal/tsd_tls.h \
jemalloc/include/jemalloc/internal/tsd_types.h \
jemalloc/include/jemalloc/internal/tsd_win.h \
jemalloc/include/jemalloc/internal/util.h \
jemalloc/include/jemalloc/internal/witness.h \
jemalloc/include/jemalloc/jemalloc.sh \
jemalloc/include/jemalloc/jemalloc_defs.h.in \
jemalloc/include/jemalloc/jemalloc_macros.h.in \
jemalloc/include/jemalloc/jemalloc_mangle.sh \
jemalloc/include/jemalloc/jemalloc_protos.h.in \
jemalloc/include/jemalloc/jemalloc_rename.sh \
jemalloc/include/jemalloc/jemalloc_typedefs.h.in \
jemalloc/include/msvc_compat/C99/stdbool.h \
jemalloc/include/msvc_compat/C99/stdint.h \
jemalloc/include/msvc_compat/strings.h \
jemalloc/include/msvc_compat/windows_extra.h \
jemalloc/jemalloc.pc.in \
jemalloc/m4/ax_cxx_compile_stdcxx.m4 \
jemalloc/msvc/ReadMe.txt \
jemalloc/msvc/jemalloc_vc2015.sln \
jemalloc/msvc/projects/vc2015/jemalloc/jemalloc.vcxproj \
jemalloc/msvc/projects/vc2015/jemalloc/jemalloc.vcxproj.filters \
jemalloc/msvc/projects/vc2015/test_threads/test_threads.cpp \
jemalloc/msvc/projects/vc2015/test_threads/test_threads.h \
jemalloc/msvc/projects/vc2015/test_threads/test_threads.vcxproj \
jemalloc/msvc/projects/vc2015/test_threads/test_threads.vcxproj.filters \
jemalloc/msvc/projects/vc2015/test_threads/test_threads_main.cpp \
jemalloc/run_tests.sh \
jemalloc/scripts/gen_run_tests.py \
jemalloc/scripts/gen_travis.py \
jemalloc/src/arena.c \
jemalloc/src/background_thread.c \
jemalloc/src/base.c \
jemalloc/src/bitmap.c \
jemalloc/src/ckh.c \
jemalloc/src/ctl.c \
jemalloc/src/extent.c \
jemalloc/src/extent_dss.c \
jemalloc/src/extent_mmap.c \
jemalloc/src/hash.c \
jemalloc/src/hooks.c \
jemalloc/src/jemalloc.c \
jemalloc/src/jemalloc_cpp.cpp \
jemalloc/src/large.c \
jemalloc/src/malloc_io.c \
jemalloc/src/mutex.c \
jemalloc/src/mutex_pool.c \
jemalloc/src/nstime.c \
jemalloc/src/pages.c \
jemalloc/src/prng.c \
jemalloc/src/prof.c \
jemalloc/src/rtree.c \
jemalloc/src/spin.c \
jemalloc/src/stats.c \
jemalloc/src/sz.c \
jemalloc/src/tcache.c \
jemalloc/src/ticker.c \
jemalloc/src/tsd.c \
jemalloc/src/witness.c \
jemalloc/src/zone.c \
jemalloc/test/include/test/SFMT-alti.h \
jemalloc/test/include/test/SFMT-params.h \
jemalloc/test/include/test/SFMT-params11213.h \
jemalloc/test/include/test/SFMT-params1279.h \
jemalloc/test/include/test/SFMT-params132049.h \
jemalloc/test/include/test/SFMT-params19937.h \
jemalloc/test/include/test/SFMT-params216091.h \
jemalloc/test/include/test/SFMT-params2281.h \
jemalloc/test/include/test/SFMT-params4253.h \
jemalloc/test/include/test/SFMT-params44497.h \
jemalloc/test/include/test/SFMT-params607.h \
jemalloc/test/include/test/SFMT-params86243.h \
jemalloc/test/include/test/SFMT-sse2.h \
jemalloc/test/include/test/SFMT.h \
jemalloc/test/include/test/btalloc.h \
jemalloc/test/include/test/extent_hooks.h \
jemalloc/test/include/test/jemalloc_test.h.in \
jemalloc/test/include/test/jemalloc_test_defs.h.in \
jemalloc/test/include/test/math.h \
jemalloc/test/include/test/mq.h \
jemalloc/test/include/test/mtx.h \
jemalloc/test/include/test/test.h \
jemalloc/test/include/test/thd.h \
jemalloc/test/include/test/timer.h \
jemalloc/test/integration/MALLOCX_ARENA.c \
jemalloc/test/integration/aligned_alloc.c \
jemalloc/test/integration/allocated.c \
jemalloc/test/integration/extent.c \
jemalloc/test/integration/extent.sh \
jemalloc/test/integration/mallocx.c \
jemalloc/test/integration/mallocx.sh \
jemalloc/test/integration/overflow.c \
jemalloc/test/integration/posix_memalign.c \
jemalloc/test/integration/rallocx.c \
jemalloc/test/integration/sdallocx.c \
jemalloc/test/integration/thread_arena.c \
jemalloc/test/integration/thread_tcache_enabled.c \
jemalloc/test/integration/xallocx.c \
jemalloc/test/integration/xallocx.sh \
jemalloc/test/src/SFMT.c \
jemalloc/test/src/btalloc.c \
jemalloc/test/src/btalloc_0.c \
jemalloc/test/src/btalloc_1.c \
jemalloc/test/src/math.c \
jemalloc/test/src/mq.c \
jemalloc/test/src/mtx.c \
jemalloc/test/src/test.c \
jemalloc/test/src/thd.c \
jemalloc/test/src/timer.c \
jemalloc/test/stress/microbench.c \
jemalloc/test/test.sh.in \
jemalloc/test/unit/SFMT.c \
jemalloc/test/unit/a0.c \
jemalloc/test/unit/arena_reset.c \
jemalloc/test/unit/arena_reset_prof.c \
jemalloc/test/unit/arena_reset_prof.sh \
jemalloc/test/unit/atomic.c \
jemalloc/test/unit/background_thread.c \
jemalloc/test/unit/base.c \
jemalloc/test/unit/bit_util.c \
jemalloc/test/unit/bitmap.c \
jemalloc/test/unit/ckh.c \
jemalloc/test/unit/decay.c \
jemalloc/test/unit/decay.sh \
jemalloc/test/unit/extent_quantize.c \
jemalloc/test/unit/fork.c \
jemalloc/test/unit/hash.c \
jemalloc/test/unit/hooks.c \
jemalloc/test/unit/junk.c \
jemalloc/test/unit/junk.sh \
jemalloc/test/unit/junk_alloc.c \
jemalloc/test/unit/junk_alloc.sh \
jemalloc/test/unit/junk_free.c \
jemalloc/test/unit/junk_free.sh \
jemalloc/test/unit/mallctl.c \
jemalloc/test/unit/malloc_io.c \
jemalloc/test/unit/math.c \
jemalloc/test/unit/mq.c \
jemalloc/test/unit/mtx.c \
jemalloc/test/unit/nstime.c \
jemalloc/test/unit/pack.c \
jemalloc/test/unit/pack.sh \
jemalloc/test/unit/pages.c \
jemalloc/test/unit/ph.c \
jemalloc/test/unit/prng.c \
jemalloc/test/unit/prof_accum.c \
jemalloc/test/unit/prof_accum.sh \
jemalloc/test/unit/prof_active.c \
jemalloc/test/unit/prof_active.sh \
jemalloc/test/unit/prof_gdump.c \
jemalloc/test/unit/prof_gdump.sh \
jemalloc/test/unit/prof_idump.c \
jemalloc/test/unit/prof_idump.sh \
jemalloc/test/unit/prof_reset.c \
jemalloc/test/unit/prof_reset.sh \
jemalloc/test/unit/prof_tctx.c \
jemalloc/test/unit/prof_tctx.sh \
jemalloc/test/unit/prof_thread_name.c \
jemalloc/test/unit/prof_thread_name.sh \
jemalloc/test/unit/ql.c \
jemalloc/test/unit/qr.c \
jemalloc/test/unit/rb.c \
jemalloc/test/unit/retained.c \
jemalloc/test/unit/rtree.c \
jemalloc/test/unit/size_classes.c \
jemalloc/test/unit/slab.c \
jemalloc/test/unit/smoothstep.c \
jemalloc/test/unit/spin.c \
jemalloc/test/unit/stats.c \
jemalloc/test/unit/stats_print.c \
jemalloc/test/unit/ticker.c \
jemalloc/test/unit/tsd.c \
jemalloc/test/unit/witness.c \
jemalloc/test/unit/zero.c \
jemalloc/test/unit/zero.sh \
build_jemalloc.sh \
# end
dist_doc_DATA = README COPYING VERSION
dist_man_MANS = man/memkind-hbw-nodes.1 \
man/hbwmalloc.3 \
man/memkind.3 \
man/memkind_arena.3 \
man/memkind_default.3 \
man/memkind_hbw.3 \
man/memkind_hugetlb.3 \
man/memkind_pmem.3 \
man/hbwallocator.3 \
man/pmemallocator.3 \
man/autohbw.7 \
# end
CLEANFILES = memkind-$(VERSION).spec
DISTCLEANFILES = VERSION
bin_PROGRAMS = memkind-hbw-nodes
memkind_hbw_nodes_SOURCES = src/memkind-hbw-nodes.c
memkind_hbw_nodes_LDADD = libmemkind.la
bin_SCRIPTS =
check_PROGRAMS =
noinst_PROGRAMS =
noinst_LTLIBRARIES =
noinst_HEADERS =
TESTS =
req_flags = -fvisibility=hidden -Wall -Werror -msse4.2 -D_GNU_SOURCE -DMEMKIND_INTERNAL_API -DJE_PREFIX=$(JE_PREFIX)
AM_CFLAGS = $(req_flags)
AM_CXXFLAGS = $(req_flags)
.PHONY: checkprogs
# create libtool .lo files from jemalloc objects
$(jemalloc_objects): jemalloc/obj/src/%.lo: jemalloc/obj/src/%.pic.o jemalloc/obj/src/%.o
@echo "# $@ - a libtool object file" > $@
@echo "# Generated by "`libtool --version | head -n 1` >> $@
@echo "# Actually generated by memkind build system spoofing libtool" >> $@
@echo "pic_object='$*.pic.o'" >> $@
@echo "non_pic_object='$*.o'" >> $@
# build check programs without running tests
checkprogs: libmemkind.la $(check_PROGRAMS)
memkind-$(VERSION).spec:
$(MAKE) version="$(VERSION)" -f memkind.spec.mk $@
rpm: dist
$(MAKE) version="$(VERSION)" CPPFLAGS="$(CPPFLAGS)" LDFLAGS="$(LDFLAGS)" \
-f memkind.spec.mk $@
all: static_lib
# the script merge memkind and jemalloc libraries into one static library.
define ar_prog
create libmemkind.a\n\
addlib .libs/libmemkind.a\n\
addlib jemalloc/obj/lib/libjemalloc_pic.a\n\
save\n\
end
endef
static_lib: jemalloc/obj/lib/libjemalloc_pic.a libmemkind.la
bash -c "ar -M < <(echo -e '$(ar_prog)')"
cp libmemkind.a .libs/
rm libmemkind.a
include autohbw/Makefile.mk
include test/Makefile.mk
include examples/Makefile.mk
include src/Makefile.mk
+42
View File
@@ -0,0 +1,42 @@
OPEN DEVELOPMENT
================
We are moving to a more open development model with memkind.
- The memkind github organization:
https://github.com/memkind
- Rather than posting bulk patches at time of tag, we are posting more
incremental patches with finer grain commit details.
- There is now a "dev" branch in the memkind repository and a
"memkind-dev" branch in the memkind fork of the jemalloc repository.
These will be used for open development work.
- There is a memkind web page here:
http://memkind.github.io/memkind
- We have posted an architecture document here:
http://memkind.github.io/memkind/memkind_arch_20150318.pdf
- We invite you to join the memkind mailing list:
https://lists.01.org/mailman/listinfo/memkind
- Feel free to open a github issue, especially if there is a TODO item
that you would like to prioritize.
- We have posted source and binary RPMs for several distributions here:
http://download.opensuse.org/repositories/home:/cmcantalupo/
- memkind is also avaiable in Fedora package repository (devel, EPEL7):
https://admin.fedoraproject.org/pkgdb/package/memkind/
- Pull requests are welcome.
- White space formatting generally conforms to
$ astyle --style=linux --indent=spaces=4 -S --max-continuation-indent=80 \
--max-code-length=80 --break-after-logical --indent-namespaces -z2 \
--align-pointer=name
- For more information about astyle:
http://astyle.sourceforge.net/
+29
View File
@@ -0,0 +1,29 @@
<!--- Provide a general summary of your changes in the Title above -->
### Description
<!--- Describe your changes in detail -->
<!--- If it fixes an open issue, please link to the issue here. -->
### Types of changes
<!--- Put an `x` in the box(es) that apply -->
- [ ] Bugfix (non-breaking change which fixes issue linked in Description above)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation (correction or otherwise)
- [ ] Cosmetics (whitespace, appearance)
- [ ] Other
### Checklist
<!--- Put an `x` in the box(es) that apply -->
- [ ] Code compiles without errors
- [ ] All tests pass locally with my changes (see TESTING section in CONTRIBUTING file)
- [ ] Created tests which will fail without the change (if possible)
- [ ] Extended the README/documentation (if necessary)
- [ ] All newly added files have proprietary license (if necessary)
- [ ] All newly added files are referenced in MANIFEST files (if necessary)
## Further comments
<!--- If this is a relatively large or complex change, kick off the discussion by explaining why you -->
<!--- choose the solution you did and what alternatives you considered, etc... -->
+251
View File
@@ -0,0 +1,251 @@
MEMKIND
=======
[![Build Status](https://travis-ci.org/memkind/memkind.svg)](https://travis-ci.org/memkind/memkind)
[![MEMKIND version](https://img.shields.io/github/tag/memkind/memkind.svg)](https://github.com/memkind/memkind/releases/latest)
[![Coverage Status](http://codecov.io/github/memkind/memkind/coverage.svg?branch=master)](http://codecov.io/gh/memkind/memkind?branch=master)
DISCLAIMER
----------
SEE COPYING FILE FOR LICENSE INFORMATION.
THIS SOFTWARE IS PROVIDED AS A DEVELOPMENT SNAPSHOT TO AID
COLLABORATION AND WAS NOT ISSUED AS A RELEASED PRODUCT BY INTEL.
LAST UPDATE
-----------
Krzysztof Kulakowski <krzysztof.kulakowski@intel.com>
Tuesday Mar 1 2016
SUMMARY
-------
The memkind library is a user extensible heap manager built on top of
jemalloc which enables control of memory characteristics and a
partitioning of the heap between kinds of memory. The kinds of memory
are defined by operating system memory policies that have been applied
to virtual address ranges. Memory characteristics supported by
memkind without user extension include control of NUMA and page size
features. The jemalloc non-standard interface has been extended to
enable specialized arenas to make requests for virtual memory from the
operating system through the memkind partition interface. Through the
other memkind interfaces the user can control and extend memory
partition features and allocate memory while selecting enabled
features.
API
---
The memkind library delivers two interfaces:
* hbwmalloc.h - recommended for high-bandwidth memory use cases (stable)
* memkind.h - generic interface for more complex use cases (partially unstable)
For more detailed information about those interfaces see
corresponding manpages (located in man/ subdir):
man memkind
or
man hbwmalloc
BUILD REQUIREMENTS
------------------
To build the memkind libraries on a RHEL Linux system first install
the required packages with the following command:
sudo yum install numactl-devel gcc-c++ unzip
On a SLES Linux system install the dependencies with the following
command:
sudo zypper install libnuma-devel gcc-c++ unzip
The memkind library uses the GNU autotools (Autoconf, Automake,
Libtool and m4) for configuration and build. The configure scripts
and gtest source code are distributed with the source tarball included
in the source RPM, and this tarball is created with the memkind "make
dist" target. In contrast to the distributed source tarball, the git
repository does not include any generated files nor the gtest source
code. For this reason some additional steps are required when
building from a checkout of the git repo. Those steps include running
the bash script called "autogen.sh" prior to configure. This script
will populate a VERSION file based on "git describe", and use
autoreconf to generate a configure script. The gtest library is
required for building the test content. This is downloaded
automatically prior to building the test content from the googlecode
website based on a target describe in memkind/test/Makefile.mk.
BUILDING
--------
a) Building jemalloc
The memkind library has a dependency on a related fork of jemalloc.
The jemalloc source was forked from jemalloc version 5.0. This source tree
is located within the jemalloc subdirectory of the memkind source. The jemalloc
source code has been kept close to the original form, and in particular
the build system has been lightly modified.
The developer must configure and build jemalloc prior to configuring
and building memkind. You can do that using included shell script:
export JE_PREFIX=jemk_
./build_jemalloc.sh
Alternatively you can follow this step-by-step instruction:
cd jemalloc
autoconf
mkdir obj
cd obj
../configure --enable-autogen --with-jemalloc-prefix=jemk_ --without-export \
--disable-stats --disable-fill \
--with-malloc-conf="narenas:256,lg_tcache_max:12"
make
cd ../..
Note: JE_PREFIX can be set to arbitrary value, including empty one.
b) Building memkind
Building and installing memkind can be as simple as typing the following while
in the root directory of the source tree:
./build.sh
make install
Alternatively you can follow this step-by-step instruction:
export JE_PREFIX=jemk_
./autogen.sh
./configure
make
make install
this should configure, build, and install this package.
See the output of:
./configure --help
for more information about either the memkind or the jemalloc
configuration options. Some useful information about building with autotools
can also be found in the INSTALL file.
**Important Notes:**
If you are using build.sh script and later want to call 'make' command directly,
then you need to call firstly:
export JE_PREFIX=jemk_
otherwise you will get an error like:
undefined reference to mallocx
RUN REQUIREMENTS
----------------
Requires kernel patch introduced in Linux v3.11 that impacts
functionality of the NUMA system calls. This is patch is commit
3964acd0dbec123aa0a621973a2a0580034b4788 in the linux-stable git
repository from kernel.org. Red Hat has back-ported this patch to the
v3.10 kernel in the RHEL 7.0 GA release, so RHEL 7.0 onward supports
memkind even though this kernel version predates v3.11.
Functionality related to hugepages allocation require patches
e0ec90ee7e6f6cbaa6d59ffb48d2a7af5e80e61d and
099730d67417dfee273e9b10ac2560ca7fac7eb9 from kernel org. Without them
physical memory may end up being located on incorrect NUMA node.
Applications using the memkind library require that libnuma and
libpthread be available for dynamic linking at run time. Both of
these libraries should be available by default, but they can be
installed on RHEL Linux with the following command:
sudo yum install pthread numactl
and on a SLES Linux system with:
sudo zypper install pthread libnuma
To use the interfaces for obtaining 2MB pages please be sure
to follow the instructions here:
https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
and pay particular attention to the use of the procfs files:
/proc/sys/vm/nr_hugepages
/proc/sys/vm/nr_overcommit_hugepages
for enabling the kernel's huge page pool.
To use the file-backed kind of memory (PMEM), please be sure that
filesystem which is used for PMEM kind supports FALLOC_FL_PUNCH_HOLE flag:
http://man7.org/linux/man-pages/man2/fallocate.2.html
SETTING LOGGING MECHANISM
-------------------------
In memkind library logging mechanism could be enabled by setting MEMKIND_DEBUG
environment variable. Setting MEMKIND_DEBUG to "1" enables printing messages
like errors and general information about environment to stderr.
SETTING HEAP MANAGER
--------------------
In memkind library heap management can be adjusted with MEMKIND_HEAP_MANAGER
environment variable, which allows for switching to one of the available
heap managers.
Values:
JEMALLOC – sets the jemalloc heap manager
TBB – sets Intel Threading Building Blocks heap manager. This option requires installed
Intel Threading Building Blocks library.
If the MEMKIND_HEAP_MANAGER is not set then the jemalloc heap manager will be used by default.
TESTING
-------
All existing tests pass. For more information on how to execute tests
see the CONTRIBUTING file.
When tests are run on a NUMA platform without high bandwidth memory
the MEMKIND_HBW_NODES environment variable is used in conjunction with
"numactl --membind" to force standard allocations to one NUMA node and
high bandwidth allocations through a different NUMA node. See next
section for more details.
SIMULATE HIGH BANDWIDTH MEMORY
------------------------------
A method for testing for the benefit of high bandwidth memory on a
dual socket Intel(R) Xeon(TM) system is to use the QPI bus to simulate
slow memory. This is not an accurate model of the bandwidth and
latency characteristics of the Intel's 2nd generation Intel(R) Xeon Phi(TM)
Product Family on package memory, but is a reasonable way to determine
which data structures rely critically on bandwidth.
If the application a.out has been modified to use high bandwidth
memory with the memkind library then this can be done with numactl as
follows with the bash shell:
export MEMKIND_HBW_NODES=0
numactl --membind=1 --cpunodebind=0 a.out
or with csh:
setenv MEMKIND_HBW_NODES 0
numactl --membind=1 --cpunodebind=0 a.out
The MEMKIND_HBW_NODES environment variable set to zero will bind high
bandwidth allocations to NUMA node 0. The --membind=1 flag to numactl
will bind standard allocations, static and stack variables to NUMA
node 1. The --cpunodebind=0 option to numactl will bind the process
threads to CPUs associated with NUMA node 0. With this configuration
standard allocations will be fetched across the QPI bus, and high
bandwidth allocations will be local to the process CPU.
NOTES
-----
* Using memkind with Transparent Huge Pages enabled may result in
undesirably high memory footprint. To avoid that disable THP using following
instruction: https://www.kernel.org/doc/Documentation/vm/transhuge.txt
STATUS
------
Different interfaces can represent different maturity level
(as described in corresponding man pages).
Feedback on design and implementation is greatly appreciated.
+1
View File
@@ -0,0 +1 @@
README
Vendored Executable
+51
View File
@@ -0,0 +1,51 @@
#!/bin/bash
#
# Copyright (C) 2018 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
ASTYLE_MIN_VER="3.1"
ASTYLE_OPT="--style=linux --indent=spaces=4 -S -r --max-continuation-indent=80 "
ASTYLE_OPT+="--max-code-length=80 --break-after-logical --indent-namespaces -z2 "
ASTYLE_OPT+="--align-pointer=name"
if ! ASTYLE=$(which astyle)
then
echo "Package astyle was not found. Unable to check source files format policy." >&2
exit 1
fi
ASTYLE_VER=$(astyle --version 2>&1 | awk '{print $NF}')
if (( $(echo "$ASTYLE_VER < $ASTYLE_MIN_VER" | bc -l) ));
then
echo "Minimal required version of astyle is $ASTYLE_MIN_VER. Detected version is $ASTYLE_VER" >&2
echo "Unable to check source files format policy." >&2
exit 1
fi
$ASTYLE $ASTYLE_OPT ./*.c,*.cpp,*.h,*.hpp --exclude=jemalloc > astyle.out
if TEST=$(cat astyle.out | grep -c Formatted)
then
cat astyle.out
git --no-pager diff
echo "Please fix style issues as shown above"
exit 1
else
echo "Source code is compliant with format policy. No style issues were found."
exit 0
fi
Vendored Executable
+48
View File
@@ -0,0 +1,48 @@
#!/bin/bash
#
# Copyright (C) 2014-2017 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
set -e
# If the VERSION file does not exist, then create it based on git
# describe or if not in a git repo just set VERSION to 0.0.0.
if [ ! -f VERSION ]; then
if [ -f .git/config ]; then
sha=$(git describe --long --always | awk -F- '{print $(NF)}')
release=$(git describe --long | awk -F- '{print $(NF-1)}')
version=$(git describe --long | sed -e "s|\(.*\)-$release-$sha|\1|" -e "s|-|+|g" -e "s|^v||")
if [ "$release" == "" ]; then
version=${sha}
else
version=${version}+dev${release}-${sha}
fi
else
echo "WARNING: VERSION file does not exist and working directory is not a git repository, setting verison to 0.0.0" 2>&1
version=0.0.0
fi
echo $version > VERSION
fi
autoreconf --install
+37
View File
@@ -0,0 +1,37 @@
#
# Copyright (C) 2014 - 2016 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
lib_LTLIBRARIES += autohbw/libautohbw.la \
# end
EXTRA_DIST += autohbw/autohbw_get_src_lines.pl
autohbw_libautohbw_la_LIBADD = libmemkind.la
autohbw_libautohbw_la_SOURCES = autohbw/autohbw.c
clean-local: autohbw-clean
autohbw-clean:
rm -f autohbw/*.gcno
+457
View File
@@ -0,0 +1,457 @@
/*
* Copyright (C) 2015 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
///////////////////////////////////////////////////////////////////////////
// File : autohbw.c
// Purpose: Library to automatically allocate HBW (MCDRAM)
// Author : Ruchira Sasanka (ruchira DOT sasanka AT intel DOT com)
// Date : Jan 30, 2015
///////////////////////////////////////////////////////////////////////////
#include <memkind.h>
#include <assert.h>
#include <ctype.h>
#include <errno.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#define AUTOHBW_EXPORT __attribute__((visibility("default")))
#define AUTOHBW_INIT __attribute__((constructor))
//-2 = nothing is printed
//-1 = critical messages are printed
// 0 = no log messages for allocations are printed but INFO messages are printed
// 1 = a log message is printed for each allocation (Default)
// 2 = a log message is printed for each allocation with a backtrace
enum {
ALWAYS = -1,
INFO,
ALLOC,
VERBOSE
};
// Default is to print allocations
static int LogLevel = ALLOC;
// Allocations of size greater than low limit promoted to HBW memory.
// If there is a high limit specified, allocations larger than this limit
// will not be allocated in HBW.
static size_t HBWLowLimit = 1 * 1024 * 1024;
static size_t HBWHighLimit = -1ull;
// Whether we have initialized HBW arena of memkind library -- by making
// a dummy call to it. HBW arena (and hence any memkind_* call with kind
// HBW) must NOT be used until this flag is set true.
static bool MemkindInitDone = false;
// Following is the type of HBW memory that is allocated using memkind.
// By changing this type, this library can be used to allocate other
// types of memory types (e.g., MEMKIND_HUGETLB, MEMKIND_GBTLB,
// MEMKIND_HBW_HUGETLB etc.)
static memkind_t hbw_kind;
// API control for HBW allocations.
static bool isAutoHBWEnabled = true;
#define LOG(level, ...) \
do { \
if (LogLevel >= level) { \
fprintf(stderr, __VA_ARGS__); \
} \
} while (0)
static bool isAllocInHBW(size_t size)
{
if (!MemkindInitDone)
return false;
if (!isAutoHBWEnabled)
return false;
if (size < HBWLowLimit)
return false;
if (size > HBWHighLimit)
return false;
return true;
}
// Returns the limit in bytes using a limit value and a multiplier
// character like K, M, G
static size_t getLimit(size_t limit, char lchar)
{
// Now read the trailing character (e.g., K, M, G)
// Based on the character, determine the multiplier
if ((limit > 0) && isalpha(lchar)) {
long mult = 1;
switch (toupper(lchar)) {
case 'G':
mult *= 1024;
case 'M':
mult *= 1024;
case 'K':
mult *= 1024;
}
// check for overflow, saturate at max
if (limit >= -1ull / mult)
return -1ull;
return limit * mult;
}
return limit;
}
// Once HBWLowLimit (and HBWHighLimit) are set, call this method to
// inform the user about the size range of arrays that will be allocated
// in HBW
static void printLimits()
{
// Inform according to the limits set
if ((HBWLowLimit > 0) && (HBWHighLimit < -1ull)) {
// if both high and low limits are specified, we use a range
LOG(INFO, "INFO: Allocations between %ldK - %ldK will be allocated in "
"HBW. Set AUTO_HBW_SIZE=X:Y to change this limit.\n",
HBWLowLimit / 1024, HBWHighLimit / 1024);
} else if (HBWLowLimit > 0) {
// if only a low limit is provided, use that
LOG(INFO, "INFO: Allocations greater than %ldK will be allocated in HBW."
" Set AUTO_HBW_SIZE=X:Y to change this limit.\n",
HBWLowLimit / 1024);
} else if (HBWHighLimit < -1ull) {
// if only a high limit is provided, use that
LOG(INFO, "INFO: Allocations smaller than %ldK will be allocated in HBW. "
"Set AUTO_HBW_SIZE=X:Y to change this limit.\n",
HBWHighLimit / 1024);
} else {
// none of limits is set to non-edge value, everything goes to HBW
LOG(INFO, "INFO: All allocation will be done in HBW.");
}
}
struct kind_name_t {
memkind_t *kind;
const char *name;
};
static struct kind_name_t named_kinds[] = {
{ &MEMKIND_DEFAULT, "memkind_default" },
{ &MEMKIND_HUGETLB, "memkind_hugetlb" },
{ &MEMKIND_INTERLEAVE, "memkind_interleave" },
{ &MEMKIND_HBW, "memkind_hbw" },
{ &MEMKIND_HBW_PREFERRED, "memkind_hbw_preferred" },
{ &MEMKIND_HBW_HUGETLB, "memkind_hbw_hugetlb" },
{ &MEMKIND_HBW_PREFERRED_HUGETLB, "memkind_hbw_preferred_hugetlb" },
{ &MEMKIND_HBW_GBTLB, "memkind_hbw_gbtlb" },
{ &MEMKIND_HBW_PREFERRED_GBTLB, "memkind_hbw_preferred_gbtlb" },
{ &MEMKIND_GBTLB, "memkind_gbtlb" },
{ &MEMKIND_HBW_INTERLEAVE, "memkind_hbw_interleave" },
};
static memkind_t get_kind_by_name(const char *name)
{
int i;
for (i = 0; i < sizeof(named_kinds) / sizeof(named_kinds[0]); ++i)
if (strcasecmp(named_kinds[i].name, name) == 0)
return *named_kinds[i].kind;
return 0;
}
// Read from the environment and sets global variables
// Env variables are:
// AUTO_HBW_SIZE = gives the size for auto HBW allocation
// AUTO_HBW_LOG = gives logging level
static void setEnvValues()
{
// STEP: Read the log level from the env variable. Do this early because
// printing depends on this
char *log_str = getenv("AUTO_HBW_LOG");
if (log_str && strlen(log_str)) {
int level = atoi(log_str);
LogLevel = level;
LOG(ALWAYS, "INFO: Setting log level to %d\n", LogLevel);
}
if (LogLevel == INFO) {
LOG(INFO, "INFO: HBW allocation stats will not be printed. "
"Set AUTO_HBW_LOG to enable.\n");
} else if (LogLevel == ALLOC) {
LOG(INFO, "INFO: Only HBW allocations will be printed. "
"Set AUTO_HBW_LOG to disable/enable.\n");
} else if (LogLevel == VERBOSE) {
LOG(INFO, "INFO: HBW allocation with backtrace info will be printed. "
"Set AUTO_HBW_LOG to disable.\n");
}
// Set the memory type allocated by this library. By default, it is
// MEMKIND_HBW, but we can use this library to allocate other memory
// types
const char *memtype_str = getenv("AUTO_HBW_MEM_TYPE");
if (memtype_str && strlen(memtype_str)) {
// Find the memkind_t using the name the user has provided in the env variable
memkind_t mty = get_kind_by_name(memtype_str);
if (mty != 0) {
hbw_kind = mty;
LOG(INFO, "INFO: Setting HBW memory type to %s\n", memtype_str);
} else {
LOG(ALWAYS, "WARN: Memory type %s not recognized. Using default type\n",
memtype_str);
}
}
// STEP: Set the size limits (thresholds) for HBW allocation
//
// Reads the environment variable
const char *size_str = getenv("AUTO_HBW_SIZE");
if (size_str) {
size_t lowlim = HBWLowLimit / 1024;
size_t highlim = HBWHighLimit / 1024;
char lowC = 'K', highC = 'K';
if (size_str) {
char *ptr = (char *)size_str;
lowlim = strtoll(ptr, &ptr, 10);
if (*ptr != 0 && *ptr != ':')
lowC = *ptr++;
else
lowC = ' ';
if (*ptr++ == ':') {
highlim = strtoll(ptr, &ptr, 10);
if (*ptr)
highC = *ptr;
else
highC = ' ';
}
LOG(INFO, "INFO: lowlim=%zu(%c), highlim=%zu(%c)\n", lowlim, lowC, highlim,
highC);
}
HBWLowLimit = getLimit(lowlim, lowC);
HBWHighLimit = getLimit(highlim, highC);
if (HBWLowLimit >= HBWHighLimit) {
LOG(ALWAYS, "WARN: In AUTO_HBW_SIZE=X:Y, X cannot be greater or equal to Y. "
"None of allocations will use HBW memory.\n");
}
} else {
// if the user did not specify any limits, inform that we are using
// default limits
LOG(INFO, "INFO: Using default values for array size thresholds. "
"Set AUTO_HBW_SIZE=X:Y to change.\n");
}
// inform the user about limits
printLimits();
}
// This function is executed at library load time.
// Initialize HBW arena by making a dummy allocation/free at library load
// time. Until HBW initialization is complete, we must not call any
// allocation routines with HBW as kind.
static void AUTOHBW_INIT autohbw_load(void)
{
// First set the default memory type this library allocates. This can
// be overridden by env variable
// Note: 'memkind_hbw_preferred' will allow falling back to DDR but
// 'memkind_hbw will not'
// Note: If HBM is not installed on a system, memkind_hbw_preferred call
// would fail. Therefore, we need to check for availability first.
if (memkind_check_available(MEMKIND_HBW) != 0) {
LOG(ALWAYS, "WARN: *** No HBM found in system. Will use default (DDR) "
"OR user specified type ***\n");
hbw_kind = MEMKIND_DEFAULT;
} else {
hbw_kind = MEMKIND_HBW_PREFERRED;
}
// Read any env variables. This has to be done first because DbgLevel
// is set using env variables and debug printing is used below
setEnvValues(); // read any env variables
LOG(INFO, "INFO: autohbw.so loaded!\n");
// dummy HBW call to initialize HBW arena
void *pp = memkind_malloc(hbw_kind, 16);
if (pp == 0) {
LOG(ALWAYS, "\t-HBW init call FAILED. "
"Is required memory type present on your system?\n");
abort();
}
LOG(ALWAYS, "\t-HBW int call succeeded\n");
memkind_free(hbw_kind, pp);
MemkindInitDone = true; // enable HBW allocation
}
static void *MemkindMalloc(size_t size)
{
LOG(VERBOSE, "In my memkind malloc sz:%ld ... ", size);
bool useHbw = isAllocInHBW(size);
memkind_t kind = useHbw ? hbw_kind : MEMKIND_DEFAULT;
if (useHbw)
LOG(VERBOSE, "\tHBW");
void *ptr = memkind_malloc(kind, size);
LOG(VERBOSE, "\tptr:%p\n", ptr);
return ptr;
}
static void *MemkindCalloc(size_t nmemb, size_t size)
{
LOG(VERBOSE, "In my memkind calloc sz:%ld ..", size * nmemb);
bool useHbw = isAllocInHBW(size);
memkind_t kind = useHbw ? hbw_kind : MEMKIND_DEFAULT;
if (useHbw)
LOG(VERBOSE, "\tHBW");
void *ptr = memkind_calloc(kind, nmemb, size);
LOG(VERBOSE, "\tptr:%p\n", ptr);
return ptr;
}
static void *MemkindRealloc(void *ptr, size_t size)
{
LOG(VERBOSE, "In my memkind realloc sz:%ld, p1:%p ..", size, ptr);
bool useHbw = isAllocInHBW(size);
memkind_t kind = useHbw ? hbw_kind : MEMKIND_DEFAULT;
if (useHbw)
LOG(VERBOSE, "\tHBW");
void *nptr = memkind_realloc(kind, ptr, size);
LOG(VERBOSE, "\tptr=%p\n", nptr);
return nptr;
}
static int MemkindAlign(void **memptr, size_t alignment, size_t size)
{
LOG(VERBOSE, "In my memkind align sz:%ld .. ", size);
bool useHbw = isAllocInHBW(size);
memkind_t kind = useHbw ? hbw_kind : MEMKIND_DEFAULT;
if (useHbw)
LOG(VERBOSE, "\tHBW");
int ret = memkind_posix_memalign(kind, memptr, alignment, size);
LOG(VERBOSE, "\tptr:%p\n", *memptr);
return ret;
}
// memkind_free does not need the exact kind, if kind is 0. Then
// the library can figure out the proper kind itself.
static void MemkindFree(void *ptr)
{
// avoid to many useless logs
if (ptr)
LOG(VERBOSE, "In my memkind free, ptr:%p\n", ptr);
memkind_free(0, ptr);
}
//--------------------------------------------------------------------------
// ------------------ Public API of autohbw ----------------------
//--------------------------------------------------------------------------
AUTOHBW_EXPORT void enableAutoHBW()
{
isAutoHBWEnabled = true;
LOG(INFO, "INFO: HBW allocations enabled by application (for this rank)\n");
}
AUTOHBW_EXPORT void disableAutoHBW()
{
isAutoHBWEnabled = false;
LOG(INFO, "INFO: HBW allocations disabled by application (for this rank)\n");
}
AUTOHBW_EXPORT void *malloc(size_t size)
{
return MemkindMalloc(size);
}
AUTOHBW_EXPORT void *calloc(size_t nmemb, size_t size)
{
return MemkindCalloc(nmemb, size);
}
AUTOHBW_EXPORT void *realloc(void *ptr, size_t size)
{
return MemkindRealloc(ptr, size);
}
AUTOHBW_EXPORT int posix_memalign(void **memptr, size_t alignment, size_t size)
{
return MemkindAlign(memptr, alignment, size);
}
// Warn about deprecated function usage.
AUTOHBW_EXPORT void *valloc(size_t size)
{
LOG(ALWAYS, "use of deprecated valloc. Use posix_memalign instead\n");
void *memptr = 0;
size_t boundary = sysconf(_SC_PAGESIZE);
int status = MemkindAlign(&memptr, boundary, size);
if (status == 0 && memptr != 0)
return memptr;
return 0;
}
// Warn about deprecated function usage.
AUTOHBW_EXPORT void *memalign(size_t boundary, size_t size)
{
LOG(ALWAYS, "use of deprecated memalign. Use posix_memalign instead\n");
void *memptr = 0;
int status = MemkindAlign(&memptr, boundary, size);
if (status == 0 && memptr != 0)
return memptr;
return 0;
}
AUTOHBW_EXPORT void free(void *ptr)
{
return MemkindFree(ptr);
}
+92
View File
@@ -0,0 +1,92 @@
README for auothbw library
Ruchira Sasanka (ruchira DOT sasanka AT intel DOT com)
2015 April 8
PURPOSE
-------
The autohbw library is used to automatically allocate high-bandwidth (HBW)
memory without any modifications to source code of your application.
This library intercepts the standard heap allocations (e.g., malloc, calloc)
in your program so that it can serve those allocations out of HBW memory.
There are two libraries installed at the same location as memkind library:
1) libautohbw.so -- dynamic library
2) libautohbw.a -- static library (provided for static linking)
USAGE
-----
To use the dynamic library, insert LD_PRELOAD=libautohbw.so and libmekind.so
before your commandline. For instance,
LD_PRELOAD=libautohbw.so /bin/ls
executes Unix ls utility with autohbw library. You can execute the
above command on the prompt or put it in a shell script. An example of
the latter approach is given in ./autohbw_test.sh. Make sure that
LD_LIBRARY_PATH enviornment variable contains the path to libautohbw.so
and libmekind.so before you exeute the above command.
To use the statically linked library, insert -lautohbw -lmemkind
on your link line. You will have to provide the library path with -L to
point to the above libraries. If you are linking your program statically
(e.g., with -static or -fast flags in ifort/icc), you must use the static
version of autohbw library (libautohbw.a).
It is possible to temporarily disable/enable automatic HBM allocations by
calling disableAutoHBW() and enableAutoHBW() in your source code. To call
these routines, please include autohbw_api.h header file and link with
-lautohbw.
Usage with MPI programs
-----------------------
To use with MPI programs, you need to create a shell script to do the
LD_PRELOAD. E.g.,
mpirun -n 2 ./autohbw_test.sh
ENVIORNMENT VARIABLES
---------------------
The following enviornment variables control the behavior of the autohbw
library:
AUTO_HBW_SIZE=x[:y]
Indicates that any allocation larger than x and smaller than y should be
allocated in HBW memory. x and y can be followed by a K, M, or G to
indicate the size in Kilo/Memga/Giga bytes (e.g., 4K, 3M, 2G).
Examples:
AUTO_HBW_SIZE=4K # all allocation larger than 4K allocated in HBW
AUTO_HBW_SIZE=1M:5M # allocations betwen 1M and 5M allocated in HBW
AUTO_HBW_LOG=level
Sets the value of logging (printing) level. If level is:
-1 = no messages are printed
0 = no log messages for allocations are printed but INFO messages
are printed
1 = a log message is printed for each allocation (Default)
2 = a log message is printed for each allocation with a backtrace
Redirect this output and use autohbw_get_src_lines.pl to find
source lines for each allocation. Your application must be compiled
with -g to see source lines.
Notes:
1. Logging adds extra overhead. Therefore, performance critical runs,
logging level should be 0
2. The amount of free memory printed with log messages is only
approximate -- e.g. pages that are not touched yet are excluded
Examples:
AUTO_HBW_LOG=1
AUTO_HBW_MEM_TYPE=memory_type
Sets the type of memory type that should be automatically allocated. By
default, this type is MEMKIND_HBW_PREFERRED, if MCDRAM is found in your
system; otherwise, the default is MEMKIND_DEFAULT. The names of memory
types are definedin memkind(3) man page. If you are requesting any huge
TLB pages, pleasemake sure that the requested type is currently enabled
in your OS.
Examples:
AUTO_HBW_MEM_TYPE=MEMKIND_HBW_PREFERRED (default, if MCDRAM present)
AUTO_HBW_MEM_TYPE=MEMKIND_DEFAULT (default, if MCDRAM absent)
AUTO_HBW_MEM_TYPE=MEMKIND_HBW_HUGETLB
AUTO_HBW_MEM_TYPE=MEMKIND_HUGETLB
+49
View File
@@ -0,0 +1,49 @@
/*
* Copyright (C) 2015 - 2016 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
///////////////////////////////////////////////////////////////////////////
// File : autohbw_api.h
// Purpose: Header file to include, to use API calls to AutoHBW
// Author : Ruchira Sasanka (ruchira.sasanka AT intel.com)
// Date : Jan 30, 2015
///////////////////////////////////////////////////////////////////////////
#ifndef AUTOHBW_API_H
#define AUTOHBW_API_H
#ifdef __cplusplus
extern "C" {
#endif
// Temporarily enable HBM allocations
void enableAutoHBW();
// Temporarily disable HBM allocations
void disableAutoHBW();
#ifdef __cplusplus
}
#endif
#endif
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/perl
#
# Copyright (C) 2015 - 2016 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
use strict;
my $usage = "Usage: get_autohbw_srclines.pl output_log_of_AutoHBW executable";
# Check for 2 arguments
#
if (@ARGV != 2) {
print $usage, "\n";
exit;
}
# Read the command line arguments
#
my $LogF = shift @ARGV;
my $BinaryF = shift @ARGV;
&main();
sub main {
print("Info: Reading AutoHBW log from: $LogF\n");
print("Info: Binary file: $BinaryF\n");
# open the log file produced by AutoHBM and look at lines starting
# with Log
open LOGF, "grep Log $LogF |" or die "Can't open log file $LogF";
my $line;
# Read each log line
#
while ($line = <LOGF>) {
# if the line contain 3 backtrace addresses, try to find the source
# lines for them
#
if ($line =~ /^(Log:.*)Backtrace:.*0x([0-9a-f]+).*0x([0-9a-f]+).*0x([0-9a-f]+)/ ) {
# Read the pointers
#
my @ptrs;
my $pre = $1;
$ptrs[0] = $2;
$ptrs[1] = $3;
$ptrs[2] = $4;
# prints the first portion of the line
#
print $pre, "\n";
# for each of the pointers, lookup its source line using
# addr2line and print the src line(s) if found
#
my $i=0;
for ($i=0; $i < @ptrs; $i++) {
my $addr = $ptrs[$i];
open SRCL, "addr2line -e $BinaryF 0x$addr |"
or die "addr2line fail";
my $srcl = <SRCL>;
if ($srcl =~ /^\?/) {
} else {
print "\t- Src: $srcl";
}
}
}
}
}
+37
View File
@@ -0,0 +1,37 @@
#!/bin/bash
# Copyright (C) 2015 - 2016 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# The following is needed if your system does not really have HBW nodes.
# Please see 'man memkind' for a description of this variable
#
# export MEMKIND_HBW_NODES=0
# Set any autohbw env variables
#
export AUTO_HBW_SIZE=4K
# Make sure you have set LD_LIBRARY_PATH to lib directory with libautohbw.so
# and libmemkind.so
#
LD_PRELOAD=libautohbw.so:libmemkind.so /bin/ls
Vendored Executable
+48
View File
@@ -0,0 +1,48 @@
#!/bin/bash
#
# Copyright (C) 2016 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
set -e
if [ -z "$JE_PREFIX" ]; then
export JE_PREFIX=jemk_
fi
cd $(dirname $0)
EXTRA_CONF=$@
if [ ! -f ./jemalloc/obj/lib/libjemalloc_pic.a ]; then
./build_jemalloc.sh $EXTRA_CONF
fi
if [ ! -f ./configure ]; then
./autogen.sh
fi
if [ ! -f ./Makefile ]; then
./configure $EXTRA_CONF
fi
#use V=1 for full cmdlines of build
make all -j $MAKEOPTS
make checkprogs -j $MAKEOPTS
+35
View File
@@ -0,0 +1,35 @@
#!/bin/bash
#
# Copyright (C) 2016 - 2018 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
EXTRA_CONF=$@
cd jemalloc
test -e configure || autoconf
test -e obj || mkdir obj
cd obj
../configure --enable-autogen --with-jemalloc-prefix=$JE_PREFIX --without-export \
--disable-stats --disable-fill \
$EXTRA_CONF --with-malloc-conf="narenas:256,lg_tcache_max:12"
make -j`nproc`
+53
View File
@@ -0,0 +1,53 @@
/* config.h.in. Generated from configure.ac by autoheader. */
/* Upper bound for number of arenas per kind */
#undef ARENA_LIMIT_PER_KIND
/* define if the compiler supports basic C++11 syntax */
#undef HAVE_CXX11
/* Define to 1 if you have the `numa' library (-lnuma). */
#undef HAVE_LIBNUMA
/* Have PTHREAD_PRIO_INHERIT. */
#undef HAVE_PTHREAD_PRIO_INHERIT
/* Enables code for debugging */
#undef MEMKIND_DEBUG
/* Enables decorators */
#undef MEMKIND_DECORATION_ENABLED
/* Enables TLS usage for mapping arenas to threads */
#undef MEMKIND_TLS
/* TLS model attribute */
#undef MEMKIND_TLS_MODEL
/* Name of package */
#undef PACKAGE
/* Define to the address where bug reports for this package should be sent. */
#undef PACKAGE_BUGREPORT
/* Define to the full name of this package. */
#undef PACKAGE_NAME
/* Define to the full name and version of this package. */
#undef PACKAGE_STRING
/* Define to the one symbol short name of this package. */
#undef PACKAGE_TARNAME
/* Define to the home page for this package. */
#undef PACKAGE_URL
/* Define to the version of this package. */
#undef PACKAGE_VERSION
/* Define to necessary symbol if this constant uses a non-standard name on
your system. */
#undef PTHREAD_CREATE_JOINABLE
/* Version number of package */
#undef VERSION
+197
View File
@@ -0,0 +1,197 @@
#
# Copyright (C) 2014 - 2018 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.
AC_PREREQ([2.64])
AC_INIT([memkind],m4_esyscmd([tr -d '\n' < VERSION]))
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_SRCDIR([memkind.spec.mk])
AM_INIT_AUTOMAKE([-Wall -Werror foreign 1.11 silent-rules subdir-objects parallel-tests tar-pax])
AM_SILENT_RULES([yes])
# Checks for programs and libraries.
AM_PROG_AR
AC_PROG_CXX
AC_PROG_CC
AC_OPENMP
AC_CHECK_LIB(numa, numa_available, [], [AC_MSG_ERROR([libnuma is required dependency])])
AX_PTHREAD([LIBS="$PTHREAD_LIBS $LIBS" CFLAGS="$CFLAGS $PTHREAD_CFLAGS" CC="$PTHREAD_CC"],
[AC_MSG_ERROR([pthreads are required dependency])])
AM_PROG_CC_C_O
#============================tls===============================================
# Check for tls_model attribute support
SAVED_CFLAGS="${CFLAGS}"
CFLAGS="$CFLAGS -Werror"
AC_MSG_CHECKING([for tls_model attribute support])
AC_COMPILE_IFELSE([AC_LANG_PROGRAM(
[[
static __thread int __attribute__((tls_model("local-dynamic"))) x;
]],
[[
x = 1234;
]])],
[AC_MSG_RESULT([yes])
tls_model="1"],
[AC_MSG_RESULT([no])
tls_model="0"])
CFLAGS="${SAVED_CFLAGS}"
if test "x${tls_model}" = "x1" ; then
AC_DEFINE([MEMKIND_TLS_MODEL],
[__attribute__((tls_model("initial-exec")))], [TLS model attribute])
else
AC_DEFINE([MEMKIND_TLS_MODEL], [ ], [TLS model attribute])
fi
AC_ARG_ENABLE([tls],
[AS_HELP_STRING([--enable-tls], [Enable thread-local storage (__thread keyword)])],
[if test "x$enable_tls" = "xyes" ; then
enable_tls="1"
else
enable_tls="0"
fi
],
[enable_tls="0"]
)
if test "x${enable_tls}" = "x1" ; then
AC_MSG_CHECKING([for TLS support])
AC_COMPILE_IFELSE([AC_LANG_PROGRAM(
[[
__thread int x;
]], [[
x = 1234;
]])],
AC_MSG_RESULT([yes]),
AC_MSG_RESULT([no])
enable_tls="0")
fi
if test "x${enable_tls}" = "x1" ; then
AC_DEFINE([MEMKIND_TLS], [ ], [Enables TLS usage for mapping arenas to threads])
fi
AC_SUBST([enable_tls])
#============================decorators========================================
AC_ARG_ENABLE([decorators],
[AS_HELP_STRING([--enable-decorators], [Enable decorators])],
[if test "x$enable_decorators" = "xyes" ; then
enable_decorators="1"
else
enable_decorators="0"
fi
],
[enable_decorators="0"]
)
if test "x${enable_decorators}" = "x1" ; then
AC_DEFINE([MEMKIND_DECORATION_ENABLED], [ ], [Enables decorators])
fi
AC_SUBST([enable_decorators])
#============================debug=============================================
AC_ARG_ENABLE([debug],
[AS_HELP_STRING([--enable-debug], [Build debugging code and compile with -O0 -g])],
[if test "x$enable_debug" = "xno" ; then
enable_debug="0"
else
enable_debug="1"
fi
],
[enable_debug="0"]
)
if test "x$enable_debug" = "x1" ; then
AC_DEFINE([MEMKIND_DEBUG], [ ], [Enables code for debugging])
CFLAGS="$CFLAGS -O0 -g"
CXXFLAGS="$CXXFLAGS -O0 -g"
fi
AC_SUBST([enable_debug])
#============================gcov==============================================
AC_ARG_ENABLE([gcov],
[AS_HELP_STRING([--enable-gcov], [Build code with gcov instructions])],
[if test "x$enable_gcov" = "xno" ; then
enable_gcov="0"
else
enable_gcov="1"
fi
],
[enable_gcov="0"]
)
if test "x$enable_gcov" = "x1" ; then
CFLAGS="$CFLAGS -O0 -fprofile-arcs -ftest-coverage"
CXXFLAGS="$CXXFLAGS -O0 -fprofile-arcs -ftest-coverage"
fi
AC_SUBST([enable_gcov])
#============================secure_flags======================================
AC_ARG_ENABLE([secure],
[AS_HELP_STRING([--enable-secure], [Build library with security enchantments])],
[if test "x$enable_secure" = "xno" ; then
enable_secure="0"
else
enable_secure="1"
fi
],
[enable_secure="1"]
)
if test "x$enable_secure" = "x1" ; then
CFLAGS="$CFLAGS -fstack-protector"
LDFLAGS="$LDFLAGS -Wl,-z,relro,-z,now"
if test "$CFLAGS" != "${CFLAGS%-O0*}" ; then # if CFLAGS contains -O0
echo "WARNING: Could not apply FORTIFY_SOURCE=2 due to lack of optimization (-O0)"
else
CFLAGS="$CFLAGS -D_FORTIFY_SOURCE=2" #FORTITFY_SOURCE does not work with -O0 (ex. if enable_debug=1 or enable_gcov=1)
fi
fi
AC_SUBST([enable_secure])
#============================arena_limit=======================================
arena_limit=256;
AC_ARG_VAR(ARENA_LIMIT,
[Upper bound for number of arenas per kind, if set to 0 then no limit]
)
if test "$ARENA_LIMIT" != "" ; then
arena_limit=$ARENA_LIMIT;
fi
AC_DEFINE_UNQUOTED([ARENA_LIMIT_PER_KIND], $arena_limit, [Upper bound for number of arenas per kind])
#============================cxx11=============================================
AX_CXX_COMPILE_STDCXX_11([noext], [optional])
AM_CONDITIONAL([HAVE_CXX11], [test "x$HAVE_CXX11" = x1])
LT_PREREQ([2.2])
LT_INIT
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
+302
View File
@@ -0,0 +1,302 @@
.gitignore
AUTHORS
COPYING
CONTRIBUTING
INSTALL
NEWS
PULL_REQUEST_TEMPLATE.md
README
VERSION
m4/ax_cxx_compile_stdcxx.m4
m4/ax_cxx_compile_stdcxx_11.m4
m4/ax_pthread.m4
ChangeLog
.travis.yml
autohbw/autohbw_README
examples/README
test/memkind-afts.ts
test/memkind-afts-ext.ts
test/memkind-slts.ts
test/memkind-perf.ts
test/memkind-perf-ext.ts
test/memkind-pytests.ts
test/gtest_fused/gtest/gtest-all.cc
test/gtest_fused/gtest/gtest.h
jemalloc/.appveyor.yml
jemalloc/.autom4te.cfg
jemalloc/.gitattributes
jemalloc/.gitignore
jemalloc/.travis.yml
jemalloc/COPYING
jemalloc/ChangeLog
jemalloc/INSTALL.md
jemalloc/Makefile.in
jemalloc/README
jemalloc/autogen.sh
jemalloc/bin/jemalloc-config.in
jemalloc/bin/jemalloc.sh.in
jemalloc/bin/jeprof.in
jemalloc/build-aux/config.guess
jemalloc/build-aux/config.sub
jemalloc/build-aux/install-sh
jemalloc/config.stamp.in
jemalloc/configure.ac
jemalloc/doc/html.xsl.in
jemalloc/doc/jemalloc.xml.in
jemalloc/doc/manpages.xsl.in
jemalloc/doc/stylesheet.xsl
jemalloc/include/jemalloc/internal/arena_externs.h
jemalloc/include/jemalloc/internal/arena_inlines_a.h
jemalloc/include/jemalloc/internal/arena_inlines_b.h
jemalloc/include/jemalloc/internal/arena_structs_a.h
jemalloc/include/jemalloc/internal/arena_structs_b.h
jemalloc/include/jemalloc/internal/arena_types.h
jemalloc/include/jemalloc/internal/assert.h
jemalloc/include/jemalloc/internal/atomic.h
jemalloc/include/jemalloc/internal/atomic_c11.h
jemalloc/include/jemalloc/internal/atomic_gcc_atomic.h
jemalloc/include/jemalloc/internal/atomic_gcc_sync.h
jemalloc/include/jemalloc/internal/atomic_msvc.h
jemalloc/include/jemalloc/internal/background_thread_externs.h
jemalloc/include/jemalloc/internal/background_thread_inlines.h
jemalloc/include/jemalloc/internal/background_thread_structs.h
jemalloc/include/jemalloc/internal/base_externs.h
jemalloc/include/jemalloc/internal/base_inlines.h
jemalloc/include/jemalloc/internal/base_structs.h
jemalloc/include/jemalloc/internal/base_types.h
jemalloc/include/jemalloc/internal/bit_util.h
jemalloc/include/jemalloc/internal/bitmap.h
jemalloc/include/jemalloc/internal/ckh.h
jemalloc/include/jemalloc/internal/ctl.h
jemalloc/include/jemalloc/internal/extent_dss.h
jemalloc/include/jemalloc/internal/extent_externs.h
jemalloc/include/jemalloc/internal/extent_inlines.h
jemalloc/include/jemalloc/internal/extent_mmap.h
jemalloc/include/jemalloc/internal/extent_structs.h
jemalloc/include/jemalloc/internal/extent_types.h
jemalloc/include/jemalloc/internal/hash.h
jemalloc/include/jemalloc/internal/hooks.h
jemalloc/include/jemalloc/internal/jemalloc_internal_decls.h
jemalloc/include/jemalloc/internal/jemalloc_internal_defs.h.in
jemalloc/include/jemalloc/internal/jemalloc_internal_externs.h
jemalloc/include/jemalloc/internal/jemalloc_internal_includes.h
jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_a.h
jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_b.h
jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_c.h
jemalloc/include/jemalloc/internal/jemalloc_internal_macros.h
jemalloc/include/jemalloc/internal/jemalloc_internal_types.h
jemalloc/include/jemalloc/internal/jemalloc_preamble.h.in
jemalloc/include/jemalloc/internal/large_externs.h
jemalloc/include/jemalloc/internal/malloc_io.h
jemalloc/include/jemalloc/internal/mutex.h
jemalloc/include/jemalloc/internal/mutex_pool.h
jemalloc/include/jemalloc/internal/mutex_prof.h
jemalloc/include/jemalloc/internal/nstime.h
jemalloc/include/jemalloc/internal/pages.h
jemalloc/include/jemalloc/internal/ph.h
jemalloc/include/jemalloc/internal/private_namespace.sh
jemalloc/include/jemalloc/internal/private_symbols.sh
jemalloc/include/jemalloc/internal/prng.h
jemalloc/include/jemalloc/internal/prof_externs.h
jemalloc/include/jemalloc/internal/prof_inlines_a.h
jemalloc/include/jemalloc/internal/prof_inlines_b.h
jemalloc/include/jemalloc/internal/prof_structs.h
jemalloc/include/jemalloc/internal/prof_types.h
jemalloc/include/jemalloc/internal/public_namespace.sh
jemalloc/include/jemalloc/internal/public_unnamespace.sh
jemalloc/include/jemalloc/internal/ql.h
jemalloc/include/jemalloc/internal/qr.h
jemalloc/include/jemalloc/internal/rb.h
jemalloc/include/jemalloc/internal/rtree.h
jemalloc/include/jemalloc/internal/rtree_tsd.h
jemalloc/include/jemalloc/internal/size_classes.sh
jemalloc/include/jemalloc/internal/smoothstep.h
jemalloc/include/jemalloc/internal/smoothstep.sh
jemalloc/include/jemalloc/internal/spin.h
jemalloc/include/jemalloc/internal/stats.h
jemalloc/include/jemalloc/internal/stats_tsd.h
jemalloc/include/jemalloc/internal/sz.h
jemalloc/include/jemalloc/internal/tcache_externs.h
jemalloc/include/jemalloc/internal/tcache_inlines.h
jemalloc/include/jemalloc/internal/tcache_structs.h
jemalloc/include/jemalloc/internal/tcache_types.h
jemalloc/include/jemalloc/internal/ticker.h
jemalloc/include/jemalloc/internal/tsd.h
jemalloc/include/jemalloc/internal/tsd_generic.h
jemalloc/include/jemalloc/internal/tsd_malloc_thread_cleanup.h
jemalloc/include/jemalloc/internal/tsd_tls.h
jemalloc/include/jemalloc/internal/tsd_types.h
jemalloc/include/jemalloc/internal/tsd_win.h
jemalloc/include/jemalloc/internal/util.h
jemalloc/include/jemalloc/internal/witness.h
jemalloc/include/jemalloc/jemalloc.sh
jemalloc/include/jemalloc/jemalloc_defs.h.in
jemalloc/include/jemalloc/jemalloc_macros.h.in
jemalloc/include/jemalloc/jemalloc_mangle.sh
jemalloc/include/jemalloc/jemalloc_protos.h.in
jemalloc/include/jemalloc/jemalloc_rename.sh
jemalloc/include/jemalloc/jemalloc_typedefs.h.in
jemalloc/include/msvc_compat/C99/stdbool.h
jemalloc/include/msvc_compat/C99/stdint.h
jemalloc/include/msvc_compat/strings.h
jemalloc/include/msvc_compat/windows_extra.h
jemalloc/jemalloc.pc.in
jemalloc/m4/ax_cxx_compile_stdcxx.m4
jemalloc/msvc/ReadMe.txt
jemalloc/msvc/jemalloc_vc2015.sln
jemalloc/msvc/projects/vc2015/jemalloc/jemalloc.vcxproj
jemalloc/msvc/projects/vc2015/jemalloc/jemalloc.vcxproj.filters
jemalloc/msvc/projects/vc2015/test_threads/test_threads.cpp
jemalloc/msvc/projects/vc2015/test_threads/test_threads.h
jemalloc/msvc/projects/vc2015/test_threads/test_threads.vcxproj
jemalloc/msvc/projects/vc2015/test_threads/test_threads.vcxproj.filters
jemalloc/msvc/projects/vc2015/test_threads/test_threads_main.cpp
jemalloc/run_tests.sh
jemalloc/scripts/gen_run_tests.py
jemalloc/scripts/gen_travis.py
jemalloc/src/arena.c
jemalloc/src/background_thread.c
jemalloc/src/base.c
jemalloc/src/bitmap.c
jemalloc/src/ckh.c
jemalloc/src/ctl.c
jemalloc/src/extent.c
jemalloc/src/extent_dss.c
jemalloc/src/extent_mmap.c
jemalloc/src/hash.c
jemalloc/src/hooks.c
jemalloc/src/jemalloc.c
jemalloc/src/jemalloc_cpp.cpp
jemalloc/src/large.c
jemalloc/src/malloc_io.c
jemalloc/src/mutex.c
jemalloc/src/mutex_pool.c
jemalloc/src/nstime.c
jemalloc/src/pages.c
jemalloc/src/prng.c
jemalloc/src/prof.c
jemalloc/src/rtree.c
jemalloc/src/spin.c
jemalloc/src/stats.c
jemalloc/src/sz.c
jemalloc/src/tcache.c
jemalloc/src/ticker.c
jemalloc/src/tsd.c
jemalloc/src/witness.c
jemalloc/src/zone.c
jemalloc/test/include/test/SFMT-alti.h
jemalloc/test/include/test/SFMT-params.h
jemalloc/test/include/test/SFMT-params11213.h
jemalloc/test/include/test/SFMT-params1279.h
jemalloc/test/include/test/SFMT-params132049.h
jemalloc/test/include/test/SFMT-params19937.h
jemalloc/test/include/test/SFMT-params216091.h
jemalloc/test/include/test/SFMT-params2281.h
jemalloc/test/include/test/SFMT-params4253.h
jemalloc/test/include/test/SFMT-params44497.h
jemalloc/test/include/test/SFMT-params607.h
jemalloc/test/include/test/SFMT-params86243.h
jemalloc/test/include/test/SFMT-sse2.h
jemalloc/test/include/test/SFMT.h
jemalloc/test/include/test/btalloc.h
jemalloc/test/include/test/extent_hooks.h
jemalloc/test/include/test/jemalloc_test.h.in
jemalloc/test/include/test/jemalloc_test_defs.h.in
jemalloc/test/include/test/math.h
jemalloc/test/include/test/mq.h
jemalloc/test/include/test/mtx.h
jemalloc/test/include/test/test.h
jemalloc/test/include/test/thd.h
jemalloc/test/include/test/timer.h
jemalloc/test/integration/MALLOCX_ARENA.c
jemalloc/test/integration/aligned_alloc.c
jemalloc/test/integration/allocated.c
jemalloc/test/integration/extent.c
jemalloc/test/integration/extent.sh
jemalloc/test/integration/mallocx.c
jemalloc/test/integration/mallocx.sh
jemalloc/test/integration/overflow.c
jemalloc/test/integration/posix_memalign.c
jemalloc/test/integration/rallocx.c
jemalloc/test/integration/sdallocx.c
jemalloc/test/integration/thread_arena.c
jemalloc/test/integration/thread_tcache_enabled.c
jemalloc/test/integration/xallocx.c
jemalloc/test/integration/xallocx.sh
jemalloc/test/src/SFMT.c
jemalloc/test/src/btalloc.c
jemalloc/test/src/btalloc_0.c
jemalloc/test/src/btalloc_1.c
jemalloc/test/src/math.c
jemalloc/test/src/mq.c
jemalloc/test/src/mtx.c
jemalloc/test/src/test.c
jemalloc/test/src/thd.c
jemalloc/test/src/timer.c
jemalloc/test/stress/microbench.c
jemalloc/test/test.sh.in
jemalloc/test/unit/SFMT.c
jemalloc/test/unit/a0.c
jemalloc/test/unit/arena_reset.c
jemalloc/test/unit/arena_reset_prof.c
jemalloc/test/unit/arena_reset_prof.sh
jemalloc/test/unit/atomic.c
jemalloc/test/unit/background_thread.c
jemalloc/test/unit/base.c
jemalloc/test/unit/bit_util.c
jemalloc/test/unit/bitmap.c
jemalloc/test/unit/ckh.c
jemalloc/test/unit/decay.c
jemalloc/test/unit/decay.sh
jemalloc/test/unit/extent_quantize.c
jemalloc/test/unit/fork.c
jemalloc/test/unit/hash.c
jemalloc/test/unit/hooks.c
jemalloc/test/unit/junk.c
jemalloc/test/unit/junk.sh
jemalloc/test/unit/junk_alloc.c
jemalloc/test/unit/junk_alloc.sh
jemalloc/test/unit/junk_free.c
jemalloc/test/unit/junk_free.sh
jemalloc/test/unit/mallctl.c
jemalloc/test/unit/malloc_io.c
jemalloc/test/unit/math.c
jemalloc/test/unit/mq.c
jemalloc/test/unit/mtx.c
jemalloc/test/unit/nstime.c
jemalloc/test/unit/pack.c
jemalloc/test/unit/pack.sh
jemalloc/test/unit/pages.c
jemalloc/test/unit/ph.c
jemalloc/test/unit/prng.c
jemalloc/test/unit/prof_accum.c
jemalloc/test/unit/prof_accum.sh
jemalloc/test/unit/prof_active.c
jemalloc/test/unit/prof_active.sh
jemalloc/test/unit/prof_gdump.c
jemalloc/test/unit/prof_gdump.sh
jemalloc/test/unit/prof_idump.c
jemalloc/test/unit/prof_idump.sh
jemalloc/test/unit/prof_reset.c
jemalloc/test/unit/prof_reset.sh
jemalloc/test/unit/prof_tctx.c
jemalloc/test/unit/prof_tctx.sh
jemalloc/test/unit/prof_thread_name.c
jemalloc/test/unit/prof_thread_name.sh
jemalloc/test/unit/ql.c
jemalloc/test/unit/qr.c
jemalloc/test/unit/rb.c
jemalloc/test/unit/retained.c
jemalloc/test/unit/rtree.c
jemalloc/test/unit/size_classes.c
jemalloc/test/unit/slab.c
jemalloc/test/unit/smoothstep.c
jemalloc/test/unit/spin.c
jemalloc/test/unit/stats.c
jemalloc/test/unit/stats_print.c
jemalloc/test/unit/ticker.c
jemalloc/test/unit/tsd.c
jemalloc/test/unit/witness.c
jemalloc/test/unit/zero.c
jemalloc/test/unit/zero.sh
include/memkind/internal/tbb_mem_pool_policy.h
+159
View File
@@ -0,0 +1,159 @@
astyle.sh
autogen.sh
build.sh
build_jemalloc.sh
install_astyle.sh
Makefile.am
configure.ac
autohbw/Makefile.mk
autohbw/autohbw.c
autohbw/autohbw_get_src_lines.pl
autohbw/autohbw_test.sh
examples/Makefile.mk
examples/autohbw_candidates.c
examples/filter_example.c
examples/hello_hbw_example.c
examples/memkind_allocated_example.cpp
examples/memkind_allocated.hpp
examples/hello_memkind_example.c
examples/memkind_decorator_debug.c
man/memkind-hbw-nodes.1
man/autohbw.7
man/hbwmalloc.3
man/memkind.3
man/memkind_arena.3
man/memkind_default.3
man/memkind_hbw.3
man/memkind_hugetlb.3
man/hbwallocator.3
man/pmemallocator.3
memkind.spec.mk
src/heap_manager.c
src/hbwmalloc.c
src/memkind.c
src/memkind_arena.c
src/memkind_default.c
src/memkind_gbtlb.c
src/memkind_hbw.c
src/memkind_regular.c
src/memkind_hugetlb.c
src/memkind_interleave.c
src/memkind_log.c
src/memkind-hbw-nodes.c
src/tbb_wrapper.c
src/Makefile.mk
include/hbwmalloc.h
include/memkind.h
include/memkind_deprecated.h
include/hbw_allocator.h
include/pmem_allocator.h
include/memkind/internal/heap_manager.h
include/memkind/internal/memkind_arena.h
include/memkind/internal/memkind_default.h
include/memkind/internal/memkind_gbtlb.h
include/memkind/internal/memkind_hbw.h
include/memkind/internal/memkind_regular.h
include/memkind/internal/memkind_hugetlb.h
include/memkind/internal/memkind_interleave.h
include/memkind/internal/memkind_pmem.h
include/memkind/internal/memkind_private.h
include/memkind/internal/memkind_log.h
include/memkind/internal/tbb_wrapper.h
test/Makefile.mk
test/Allocator.hpp
test/TestPolicy.hpp
test/bat_tests.cpp
test/check.cpp
test/check.h
test/common.h
test/static_kinds_list.h
test/environ_err_hbw_malloc_test.cpp
test/gb_page_tests_bind_policy.cpp
test/main.cpp
test/multithreaded_tests.cpp
test/negative_tests.cpp
test/test.sh
test/trial_generator.cpp
test/trial_generator.h
test/error_message_tests.cpp
test/get_arena_test.cpp
test/locality_test.cpp
test/memkind_pmem_tests.cpp
test/memkind_pmem_long_time_tests.cpp
test/memory_footprint_test.cpp
test/memory_manager.h
test/random_sizes_allocator.h
test/proc_stat.h
test/static_kinds_tests.cpp
test/huge_page_test.cpp
test/hbw_verify_function_test.cpp
test/performance/framework.hpp
test/performance/framework.cpp
test/performance/operations.hpp
test/performance/perf_tests.hpp
test/performance/perf_tests.cpp
test/hbw_allocator_tests.cpp
test/decorator_test.cpp
test/decorator_test.h
test/alloc_performance_tests.cpp
test/allocator_perf_tool/AllocationSizes.hpp
test/allocator_perf_tool/Allocation_info.hpp
test/allocator_perf_tool/Allocation_info.cpp
test/allocator_perf_tool/Allocator.hpp
test/allocator_perf_tool/AllocatorFactory.hpp
test/allocator_perf_tool/CSVLogger.hpp
test/allocator_perf_tool/CommandLine.hpp
test/allocator_perf_tool/Configuration.hpp
test/allocator_perf_tool/ConsoleLog.hpp
test/allocator_perf_tool/FunctionCalls.hpp
test/allocator_perf_tool/FunctionCallsPerformanceTask.cpp
test/allocator_perf_tool/FunctionCallsPerformanceTask.h
test/allocator_perf_tool/GTestAdapter.hpp
test/allocator_perf_tool/Iterator.hpp
test/allocator_perf_tool/JemallocAllocatorWithTimer.hpp
test/allocator_perf_tool/Makefile
test/allocator_perf_tool/MemkindAllocatorWithTimer.hpp
test/allocator_perf_tool/Numastat.hpp
test/allocator_perf_tool/Runnable.hpp
test/allocator_perf_tool/PmemMockup.cpp
test/allocator_perf_tool/PmemMockup.hpp
test/allocator_perf_tool/ScenarioWorkload.cpp
test/allocator_perf_tool/ScenarioWorkload.h
test/allocator_perf_tool/StandardAllocatorWithTimer.hpp
test/allocator_perf_tool/Stats.hpp
test/allocator_perf_tool/StressIncreaseToMax.cpp
test/allocator_perf_tool/StressIncreaseToMax.h
test/allocator_perf_tool/Task.hpp
test/allocator_perf_tool/TaskFactory.hpp
test/allocator_perf_tool/Tests.hpp
test/allocator_perf_tool/Thread.hpp
test/allocator_perf_tool/TimerSysTime.hpp
test/allocator_perf_tool/VectorIterator.hpp
test/allocator_perf_tool/Workload.hpp
test/allocator_perf_tool/WrappersMacros.hpp
test/allocator_perf_tool/HugePageUnmap.hpp
test/allocator_perf_tool/HugePageOrganizer.hpp
test/allocator_perf_tool/HBWmallocAllocatorWithTimer.hpp
test/allocator_perf_tool/main.cpp
test/allocate_to_max_stress_test.cpp
test/memkind_versioning_tests.cpp
test/heap_manager_init_perf_test.cpp
test/autohbw_test_helper.c
test/trace_mechanism_test_helper.c
test/hbw_detection_test.py
test/autohbw_test.py
test/trace_mechanism_test.py
test/python_framework/cmd_helper.py
test/python_framework/huge_page_organizer.py
test/python_framework/__init__.py
test/draw_plots.py
test/run_alloc_benchmark.sh
test/alloc_benchmark.c
test/load_tbbmalloc_symbols.c
test/tbbmalloc.h
test/freeing_memory_segfault_test.cpp
test/dlopen_test.cpp
test/pmem_allocator_tests.cpp
man/memkind_pmem.3
src/memkind_pmem.c
+10
View File
@@ -0,0 +1,10 @@
examples/pmem_kinds.c
examples/pmem_malloc.c
examples/pmem_malloc_unlimited.c
examples/pmem_usable_size.c
examples/pmem_alignment.c
examples/pmem_and_default_kind
examples/pmem_multithreads.c
examples/pmem_multithreads_onekind.c
examples/pmem_free_with_unknown_kind.c
examples/pmem_cpp_allocator.cpp
+23
View File
@@ -0,0 +1,23 @@
/*
* Copyright (C) 2014 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
+31
View File
@@ -0,0 +1,31 @@
/*
* Copyright (c) 2015, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
+36
View File
@@ -0,0 +1,36 @@
/*
* Copyright (C) 2000 - 2014, Intel Corp.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions, and the following disclaimer,
* without modification.
* 2. Redistributions in binary form must reproduce at minimum a disclaimer
* substantially similar to the "NO WARRANTY" disclaimer below
* ("Disclaimer") and any redistribution must be conditioned upon
* including a substantially similar Disclaimer requirement for further
* binary redistribution.
* 3. Neither the names of the above-listed copyright holders nor the names
* of any contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* Alternatively, this software may be distributed under the terms of the
* GNU General Public License ("GPL") version 2 as published by the Free
* Software Foundation.
*
* NO WARRANTY
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
* STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
* IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGES.
*/
+5
View File
@@ -0,0 +1,5 @@
memkind (0.0.0) UNRELEASED; urgency=low
* See ChangeLog for more details
-- Krzysztof Kulakowski <krzysztof.kulakowski@intel.com> Mon, 12 Dec 2016 15:53:31 +0000
+1
View File
@@ -0,0 +1 @@
9
+25
View File
@@ -0,0 +1,25 @@
Source: memkind
Section: utils
Priority: optional
Maintainer: Krzysztof Kulakowski <krzysztof.kulakowski@intel.com>
Build-Depends: debhelper (>= 8.0.0)
Standards-Version: 3.9.4
Homepage: https://github.com/memkind/memkind
Package: memkind
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description:
The memkind library is an user extensible heap manager built on top of
jemalloc which enables control of memory characteristics and a
partitioning of the heap between kinds of memory. The kinds of memory
are defined by operating system memory policies that have been applied
to virtual address ranges. Memory characteristics supported by
memkind without user extension include control of NUMA and page size
features. The jemalloc non-standard interface has been extended to
enable specialized arenas to make requests for virtual memory from the
operating system through the memkind partition interface. Through the
other memkind interfaces the user can control and extend memory
partition features and allocate memory while selecting enabled
features. This software is being made available for early evaluation.
Feedback on design or implementation is greatly appreciated.
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/make -f
# -*- makefile -*-
# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1
# ignore tests during build
override_dh_auto_test:
override_dh_auto_configure:
./build_jemalloc.sh
./autogen.sh
dh_auto_configure -- --prefix=/usr --libdir=/usr/lib \
--includedir=/usr/include --sbindir=/usr/sbin --enable-cxx11 \
--mandir=/usr/share/man --docdir=/usr/share/doc/memkind
override_dh_auto_build:
dh_auto_build -- checkprogs
%:
dh $@
+85
View File
@@ -0,0 +1,85 @@
#
# Copyright (C) 2014 - 2018 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
noinst_PROGRAMS += examples/hello_memkind \
examples/hello_memkind_debug \
examples/hello_hbw \
examples/filter_memkind \
examples/pmem_kinds \
examples/pmem_malloc \
examples/pmem_malloc_unlimited \
examples/pmem_usable_size \
examples/pmem_alignment \
examples/pmem_and_default_kind \
examples/pmem_multithreads \
examples/pmem_multithreads_onekind \
examples/pmem_free_with_unknown_kind \
examples/autohbw_candidates \
# end
if HAVE_CXX11
noinst_PROGRAMS += examples/memkind_allocated
noinst_PROGRAMS += examples/pmem_cpp_allocator
endif
examples_hello_memkind_LDADD = libmemkind.la
examples_hello_memkind_debug_LDADD = libmemkind.la
examples_hello_hbw_LDADD = libmemkind.la
examples_filter_memkind_LDADD = libmemkind.la
examples_pmem_kinds_LDADD = libmemkind.la
examples_pmem_malloc_LDADD = libmemkind.la
examples_pmem_malloc_unlimited_LDADD = libmemkind.la
examples_pmem_usable_size_LDADD = libmemkind.la
examples_pmem_alignment_LDADD = libmemkind.la
examples_pmem_and_default_kind_LDADD = libmemkind.la
examples_pmem_multithreads_LDADD = libmemkind.la
examples_pmem_multithreads_onekind_LDADD = libmemkind.la
examples_pmem_free_with_unknown_kind_LDADD = libmemkind.la
examples_autohbw_candidates_LDADD = libmemkind.la
if HAVE_CXX11
examples_memkind_allocated_LDADD = libmemkind.la
examples_pmem_cpp_allocator_LDADD = libmemkind.la
endif
examples_hello_memkind_SOURCES = examples/hello_memkind_example.c
examples_hello_memkind_debug_SOURCES = examples/hello_memkind_example.c examples/memkind_decorator_debug.c
examples_hello_hbw_SOURCES = examples/hello_hbw_example.c
examples_filter_memkind_SOURCES = examples/filter_example.c
examples_pmem_kinds_SOURCES = examples/pmem_kinds.c
examples_pmem_malloc_SOURCES = examples/pmem_malloc.c
examples_pmem_malloc_unlimited_SOURCES = examples/pmem_malloc_unlimited.c
examples_pmem_usable_size_SOURCES = examples/pmem_usable_size.c
examples_pmem_alignment_SOURCES = examples/pmem_alignment.c
examples_pmem_and_default_kind_SOURCES = examples/pmem_and_default_kind.c
examples_pmem_multithreads_SOURCES = examples/pmem_multithreads.c
examples_pmem_multithreads_onekind_SOURCES = examples/pmem_multithreads_onekind.c
examples_pmem_free_with_unknown_kind_SOURCES = examples/pmem_free_with_unknown_kind.c
examples_autohbw_candidates_SOURCES = examples/autohbw_candidates.c
if HAVE_CXX11
examples_memkind_allocated_SOURCES = examples/memkind_allocated_example.cpp examples/memkind_allocated.hpp
examples_pmem_cpp_allocator_SOURCES = examples/pmem_cpp_allocator.cpp
endif
clean-local:
rm -f examples/*.gcno examples/*.gcda
+80
View File
@@ -0,0 +1,80 @@
# Memkind examples
The example directory contains example codes that use the memkind
interface.
## PMEM
The pmem_*.c(pp) demonstrates how to create and use a file-backed memory kind.
The default pmem path is "/tmp/".
Custom directory is pass as first argument to all of PMEM example programs,
e.g. to execute pmem_malloc example in /mnt/pmem location, call:
./pmem_malloc /mnt/pmem/
### pmem_kinds.c
This example shows how to create and destroy pmem kind with defined or unlimited size.
### pmem_malloc.c
This example shows how to allocate memory and possibility to exceed pmem kind size.
### pmem_malloc_unlimited.c
This example shows how to allocate memory with unlimited kind size.
### pmem_usable_size.c
This example shows difference between the expected and the actual allocation size.
### pmem_alignment.c
This example shows how to use memkind alignment and how it affects allocations.
### pmem_multithreads.c
This example shows how to use multithreading with independent pmem kinds.
### pmem_multithreads_onekind.c
This example shows how to use multithreading with one main pmem kind.
### pmem_and_default_kind.c
This example shows how to allocate in standard memory and file-backed memory (pmem kind).
### pmem_free_with_unknown_kind.c
This example shows how to allocate in standard memory and file-backed memory (pmem kind)
and free memory without a need to remember which kind it belongs to.
### pmem_cpp_allocator.cpp
This example shows usage of C++ allocator mechanism designed for file-backed memory
kind with different data structures like: vector, list and map.
## Other memkind examples
The simplest example is the hello_example.c which is a hello world
variant. The filter_example.c shows how you would use high bandwidth
memory to store a reduction of a larger data set stored in DDR. There is
also an example of how to create user defined kinds. This example
creates kinds which isolate allocations to a single NUMA node each
backed by a single arena.
The memkind_allocated example is simple usage of memkind in C++11 which
shows how memkind can be used to allocate objects, and consists of two files:
memkind_allocated.hpp - which is definition of template class that should be
inherited and parametrized by derived type (curiously recurring template
pattern), to let deriving class allocate their objects using specified kind.
memkind_allocated_example.cpp - which is usage example of this approach.
Logic of memkind_allocated is based on overriding operator new() in template,
and allocating memory on kind specified in new() parameter, or by overridable
static method getClassKind(). This implementation also supports alignment
specifier's (alignas() - new feature in C++11).
The downside of this approach is that it will work properly only if
memkind_allocated template is inherited once in inheritance chain (which
probably makes that not very useful for most scenarios). Other thing is that it
overriding class new() operator which can cause various problems if used
unwisely.
+122
View File
@@ -0,0 +1,122 @@
/*
* Copyright (C) 2015 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
///////////////////////////////////////////////////////////////////////////
// File : autohbw_candidates.c
// Purpose: Shows which functions are interposed by AutoHBW library.
// : These functions can be used for testing purposes
// Author : Ruchira Sasanka (ruchira.sasanka AT intel.com)
// Date : Sept 10, 2015
///////////////////////////////////////////////////////////////////////////
#include <memkind.h>
#include <stdlib.h>
#include <stdio.h>
///////////////////////////////////////////////////////////////////////////
// This function contains an example case for each heap allocation function
// intercepted by the AutoHBW library
///////////////////////////////////////////////////////////////////////////
//volatile is needed to prevent optimizing out below hooks
volatile int memkind_called_g;
void memkind_malloc_post(struct memkind *kind, size_t size, void **result)
{
memkind_called_g = 1;
}
void memkind_calloc_post(struct memkind *kind, size_t nmemb, size_t size,
void **result)
{
memkind_called_g = 1;
}
void memkind_posix_memalign_post(struct memkind *kind, void **memptr,
size_t alignment, size_t size, int *err)
{
memkind_called_g = 1;
}
void memkind_realloc_post(struct memkind *kind, void *ptr, size_t size,
void **result)
{
memkind_called_g = 1;
}
void memkind_free_pre(struct memkind **kind, void **ptr)
{
memkind_called_g = 1;
}
void finish_testcase(int fail_condition, const char *fail_message, int *err)
{
if(memkind_called_g != 1 || fail_condition) {
printf("%s\n", fail_message);
*err= -1;
}
memkind_called_g = 0;
}
int main()
{
int err = 0;
const size_t size = 1024 * 1024; // 1M of data
void *buf = NULL;
memkind_called_g = 0;
// Test 1: Test malloc and free
buf = malloc(size);
finish_testcase(buf==NULL, "Malloc failed!", &err);
free(buf);
finish_testcase(0, "Free after malloc failed!", &err);
// Test 2: Test calloc and free
buf = calloc(size, 1);
finish_testcase(buf==NULL, "Calloc failed!", &err);
free(buf);
finish_testcase(0, "Free after calloc failed!", &err);
// Test 3: Test realloc and free
buf = malloc(size);
finish_testcase(buf==NULL, "Malloc before realloc failed!", &err);
buf = realloc(buf, size * 2);
finish_testcase(buf==NULL, "Realloc failed!", &err);
free(buf);
finish_testcase(0, "Free after realloc failed!", &err);
// Test 4: Test posix_memalign and free
int ret = posix_memalign(&buf, 64, size);
finish_testcase(ret, "Posix_memalign failed!", &err);
free(buf);
finish_testcase(0, "Free after posix_memalign failed!", &err);
return err;
}
+87
View File
@@ -0,0 +1,87 @@
/*
* Copyright (C) 2014 - 2016 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
int main(int argc, char **argv)
{
const size_t stream_len = 1024 * 1024;
const size_t filter_len = 1024;
const size_t num_filter = stream_len / filter_len;
size_t i, j;
double *stream = NULL;
double *filter = NULL;
double *result = NULL;
srandom(0);
stream = (double *)memkind_malloc(MEMKIND_DEFAULT, stream_len * sizeof(double));
if (stream == NULL) {
perror("<memkind>");
fprintf(stderr, "Unable to allocate stream\n");
return errno ? -errno : 1;
}
filter = (double *)memkind_malloc(MEMKIND_HBW, filter_len * sizeof(double));
if (filter == NULL) {
perror("<memkind>");
fprintf(stderr, "Unable to allocate filter\n");
return errno ? -errno : 1;
}
result = (double *)memkind_calloc(MEMKIND_HBW, filter_len, sizeof(double));
if (result == NULL) {
perror("<memkind>");
fprintf(stderr, "Unable to allocate result\n");
return errno ? -errno : 1;
}
for (i = 0; i < stream_len; i++) {
stream[i] = (double)(random())/(double)(RAND_MAX);
}
for (i = 0; i < filter_len; i++) {
filter[i] = (double)(i)/(double)(filter_len);
}
for (i = 0; i < num_filter; i++) {
for (j = 0; j < filter_len; j++) {
result[j] += stream[i * filter_len + j] * filter[j];
}
}
for (i = 0; i < filter_len; i++) {
fprintf(stdout, "%.6e\n", result[i]);
}
memkind_free(MEMKIND_HBW, result);
memkind_free(MEMKIND_HBW, filter);
memkind_free(MEMKIND_DEFAULT, stream);
return 0;
}
+81
View File
@@ -0,0 +1,81 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <hbwmalloc.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
int main(int argc, char **argv)
{
const size_t size = 512;
char *default_str = NULL;
char *hbw_str = NULL;
char *hbw_hugetlb_str = NULL;
int err = 0;
default_str = (char *)malloc(size);
if (default_str == NULL) {
perror("malloc()");
fprintf(stderr, "Unable to allocate default string\n");
err = errno ? -errno : 1;
goto exit;
}
hbw_str = (char *)hbw_malloc(size);
if (hbw_str == NULL) {
perror("hbw_malloc()");
fprintf(stderr, "Unable to allocate hbw string\n");
err = errno ? -errno : 1;
goto exit;
}
err = hbw_posix_memalign_psize((void **)&hbw_hugetlb_str, 2097152, size,
HBW_PAGESIZE_2MB);
if (err) {
perror("hbw_posix_memalign()");
fprintf(stderr, "Unable to allocate hbw hugetlb string\n");
err = errno ? -errno : 1;
goto exit;
}
sprintf(default_str, "Hello world from standard memory\n");
sprintf(hbw_str, "Hello world from high bandwidth memory\n");
sprintf(hbw_hugetlb_str, "Hello world from high bandwidth 2 MB paged memory\n");
fprintf(stdout, "%s", default_str);
fprintf(stdout, "%s", hbw_str);
fprintf(stdout, "%s", hbw_hugetlb_str);
exit:
if (hbw_hugetlb_str) {
hbw_free(hbw_hugetlb_str);
}
if (hbw_str) {
hbw_free(hbw_str);
}
if (default_str) {
free(default_str);
}
return err;
}
+115
View File
@@ -0,0 +1,115 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
int main(int argc, char **argv)
{
const size_t size = 512;
char *default_str = NULL;
char *hugetlb_str = NULL;
char *hbw_str = NULL;
char *hbw_hugetlb_str = NULL;
char *hbw_preferred_str = NULL;
char *hbw_preferred_hugetlb_str = NULL;
char *hbw_interleave_str = NULL;
default_str = (char *)memkind_malloc(MEMKIND_DEFAULT, size);
if (default_str == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate default string\n");
return errno ? -errno : 1;
}
hugetlb_str = (char *)memkind_malloc(MEMKIND_HUGETLB, size);
if (hugetlb_str == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate hugetlb string\n");
return errno ? -errno : 1;
}
hbw_str = (char *)memkind_malloc(MEMKIND_HBW, size);
if (hbw_str == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate hbw string\n");
return errno ? -errno : 1;
}
hbw_hugetlb_str = (char *)memkind_malloc(MEMKIND_HBW_HUGETLB, size);
if (hbw_hugetlb_str == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate hbw_hugetlb string\n");
return errno ? -errno : 1;
}
hbw_preferred_str = (char *)memkind_malloc(MEMKIND_HBW_PREFERRED, size);
if (hbw_preferred_str == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate hbw_preferred string\n");
return errno ? -errno : 1;
}
hbw_preferred_hugetlb_str = (char *)memkind_malloc(
MEMKIND_HBW_PREFERRED_HUGETLB, size);
if (hbw_preferred_hugetlb_str == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate hbw_preferred_hugetlb string\n");
return errno ? -errno : 1;
}
hbw_interleave_str = (char *)memkind_malloc(MEMKIND_HBW_INTERLEAVE, size);
if (hbw_interleave_str == NULL) {
perror("memkind_malloc()");
fprintf(stderr,"Unable to allocate hbw_interleave string\n");
return errno ? -errno : 1;
}
sprintf(default_str, "Hello world from standard memory\n");
sprintf(hugetlb_str, "Hello world from standard memory with 2 MB pages\n");
sprintf(hbw_str, "Hello world from high bandwidth memory\n");
sprintf(hbw_hugetlb_str, "Hello world from high bandwidth 2 MB paged memory\n");
sprintf(hbw_preferred_str,
"Hello world from high bandwidth memory if sufficient resources exist\n");
sprintf(hbw_preferred_hugetlb_str,
"Hello world from high bandwidth 2 MB paged memory if sufficient resources exist\n");
sprintf(hbw_interleave_str,
"Hello world from high bandwidth interleaved memory\n");
fprintf(stdout, "%s", default_str);
fprintf(stdout, "%s", hugetlb_str);
fprintf(stdout, "%s", hbw_str);
fprintf(stdout, "%s", hbw_hugetlb_str);
fprintf(stdout, "%s", hbw_preferred_str);
fprintf(stdout, "%s", hbw_preferred_hugetlb_str);
fprintf(stdout, "%s", hbw_interleave_str);
memkind_free(MEMKIND_HBW_INTERLEAVE, hbw_interleave_str);
memkind_free(MEMKIND_HBW_PREFERRED_HUGETLB, hbw_preferred_hugetlb_str);
memkind_free(MEMKIND_HBW_PREFERRED, hbw_preferred_str);
memkind_free(MEMKIND_HBW_HUGETLB, hbw_hugetlb_str);
memkind_free(MEMKIND_HBW, hbw_str);
memkind_free(MEMKIND_HUGETLB, hugetlb_str);
memkind_free(MEMKIND_DEFAULT, default_str);
return 0;
}
+107
View File
@@ -0,0 +1,107 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
///////////////////////////////////////////////////////////////////////////////////////////////////////////
#include <memkind.h>
#include <cstdlib>
#include <new>
template <class deriving_class>
class memkind_allocated
{
public:
static memkind_t getClassKind()
{
return MEMKIND_DEFAULT;
}
void *operator new(std::size_t size)
{
return deriving_class::operator new(size, deriving_class::getClassKind());
}
void *operator new[](std::size_t size)
{
return deriving_class::operator new(size, deriving_class::getClassKind());
}
void *operator new(std::size_t size, memkind_t memory_kind)
{
void *result_ptr = NULL;
int allocation_result = 0;
//This check if deriving_class has specified alignment, which is suitable
//to be used with posix_memalign()
if(alignof(deriving_class) < sizeof(void *)) {
result_ptr = memkind_malloc(memory_kind, size);
allocation_result = result_ptr ? 1 : 0;
} else {
allocation_result = memkind_posix_memalign(memory_kind, &result_ptr,
alignof(deriving_class), size);
}
if(allocation_result) {
throw std::bad_alloc();
}
return result_ptr;
}
void *operator new[](std::size_t size, memkind_t memory_kind)
{
return deriving_class::operator new(size, memory_kind);
}
void operator delete(void *ptr, memkind_t memory_kind)
{
memkind_free(memory_kind, ptr);
}
void operator delete(void *ptr)
{
memkind_free(0, ptr);
}
void operator delete[](void *ptr)
{
deriving_class::operator delete(ptr);
}
void operator delete[](void *ptr, memkind_t memory_kind)
{
deriving_class::operator delete(ptr, memory_kind);
}
protected:
memkind_allocated()
{
}
~memkind_allocated()
{
}
};
+105
View File
@@ -0,0 +1,105 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
///////////////////////////////////////////////////////////////////////////////////////////////////////////
#include <memkind.h>
#include "memkind_allocated.hpp"
// This code is example usage of C++11 features with custom allocator,
// support for C++11 is required.
#include <iostream>
#if __cplusplus > 199711L
#include <cstdlib>
#include <new>
#include <string>
//example class definition, which derive from memkind_allocated template to
//have objects allocated with memkind, and have alignment specified by alignas()
class alignas(128) memkind_allocated_example : public
memkind_allocated<memkind_allocated_example>
{
std::string message;
public:
//Override method for returning class default kind to make objects be by default allocated on High-Bandwith Memory
static memkind_t getClassKind()
{
return MEMKIND_HBW;
}
memkind_allocated_example(std::string my_message)
{
this->message = my_message;
}
memkind_allocated_example()
{
}
void print_message()
{
std::cout << message << std::endl;
std::cout << "Memory adress of this object is: " << (void *)this << std::endl
<< std::endl;
}
};
int main()
{
memkind_t specified_kind = MEMKIND_HBW_HUGETLB;
memkind_allocated_example *default_kind_example = new memkind_allocated_example(
std::string("This object has been allocated using class default kind, which is: MEMKIND_DEFAULT") );
default_kind_example->print_message();
delete default_kind_example;
memkind_allocated_example *specified_kind_example = new(
specified_kind) memkind_allocated_example(
std::string("This object has been allocated using specified kind, which is: MEMKIND_HBW_HUGETLB") );
specified_kind_example->print_message();
delete specified_kind_example;
//examples for using same aproach for allocating arrays of objects, note that objects created that way can be initialized only with default (unparameterized) constructor
memkind_allocated_example *default_kind_array_example = new
memkind_allocated_example[5]();
delete[] default_kind_array_example;
memkind_allocated_example *specified_kind_array_example = new(
specified_kind) memkind_allocated_example[5]();
delete[] specified_kind_array_example;
return 0;
}
#else //If C++11 is not avaiable - do nothing.
int main()
{
std::cout << "WARNING: because your compiler does not support C++11 standard,"
<< std::endl;
std::cout << "this example is only as a dummy placeholder." << std::endl;
return 0;
}
#endif
+64
View File
@@ -0,0 +1,64 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdlib.h>
#include <stdio.h>
/* This is an example that enables debug printing on every alloction call */
static void memkind_debug(const char *func, memkind_t kind, size_t size,
void *ptr)
{
fprintf(stderr, "[ DEBUG ] func=%s kind=%p size=%zu ptr=0x%lx\n", func, kind,
size, (size_t)ptr);
}
void memkind_malloc_post(memkind_t kind, size_t size, void **result)
{
memkind_debug("memkind_malloc", kind, size, *result);
}
void memkind_calloc_post(memkind_t kind, size_t nmemb, size_t size,
void **result)
{
memkind_debug("memkind_calloc", kind, nmemb * size, *result);
}
void memkind_posix_memalign_post(memkind_t kind, void **memptr,
size_t alignment, size_t size, int *err)
{
memkind_debug("memkind_posix_memalign", kind, size, *memptr);
}
void memkind_realloc_post(memkind_t kind, void *ptr, size_t size, void **result)
{
memkind_debug("memkind_realloc", kind, size, *result);
}
void memkind_free_pre(memkind_t kind, void **ptr)
{
memkind_debug("memkind_free", kind, 0, *ptr);
}
+136
View File
@@ -0,0 +1,136 @@
/*
* Copyright (c) 2018 Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdio.h>
#include <errno.h>
#include <sys/stat.h>
#define PMEM_MAX_SIZE (1024 * 1024 * 32)
static char *PMEM_DIR = "/tmp/";
int main(int argc, char *argv[])
{
struct memkind *pmem_kind = NULL;
int err = 0;
struct stat st;
if (argc > 2) {
fprintf(stderr, "Usage: %s [pmem_kind_dir_path]", argv[0]);
return 1;
} else if (argc == 2) {
if (stat(argv[1], &st) != 0 || !S_ISDIR(st.st_mode)) {
fprintf(stderr, "%s : Invalid path to pmem kind directory", argv[1]);
return 1;
} else {
PMEM_DIR = argv[1];
}
}
fprintf(stdout,
"This example shows how to use memkind alignment and how it affects allocations.\nPMEM kind directory: %s\n",
PMEM_DIR);
/* Create PMEM partition with PMEM_MAX_SIZE size */
err = memkind_create_pmem(PMEM_DIR, PMEM_MAX_SIZE, &pmem_kind);
if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return errno ? -errno : 1;
}
char *pmem_str10 = NULL;
char *pmem_str11 = NULL;
/* Lets make two 32 bytes allocations */
pmem_str10 = (char *)memkind_malloc(pmem_kind, 32);
if (pmem_str10 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str10)\n");
return errno ? -errno : 1;
}
pmem_str11 = (char *)memkind_malloc(pmem_kind, 32);
if (pmem_str11 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str11)\n");
return errno ? -errno : 1;
}
/* They will be very close to each other in memory */
if (pmem_str11 - pmem_str10 != 32) {
fprintf(stderr, "Something went wrong\n");
return 1;
}
memkind_free(pmem_kind, pmem_str10);
memkind_free(pmem_kind, pmem_str11);
/* Lets make two 32 bytes allocations with alignment 64 this time */
err = memkind_posix_memalign(pmem_kind, (void **)&pmem_str10, 64, 32);
if (err) {
perror("memkind_posix_memalign()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str10) with alignment\n");
return errno ? -errno : 1;
}
err = memkind_posix_memalign(pmem_kind, (void **)&pmem_str11, 64, 32);
if (err) {
perror("memkind_posix_memalign()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str11) with alignment\n");
return errno ? -errno : 1;
}
/* This time addresses are not close to each other, they are aligned to 64 */
if (pmem_str11 - pmem_str10 != 64) {
fprintf(stderr, "Something went wrong with alignment allocation\n");
return 1;
}
memkind_free(pmem_kind, pmem_str10);
memkind_free(pmem_kind, pmem_str11);
err = memkind_destroy_kind(pmem_kind);
if (err) {
perror("memkind_destroy_kind()");
fprintf(stderr, "Unable to destroy pmem partition\n");
return errno ? -errno : 1;
}
fprintf(stdout,
"The memory has been successfully allocated using memkind alignment.");
return 0;
}
+149
View File
@@ -0,0 +1,149 @@
/*
* Copyright (c) 2018 Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdio.h>
#include <errno.h>
#include <sys/resource.h>
#include <sys/stat.h>
#include <sys/time.h>
#define MB (1024 * 1024)
#define HEAP_LIMIT_SIMULATE (1024 * MB)
static char *PMEM_DIR = "/tmp/";
int main(int argc, char *argv[])
{
const size_t size = 512;
struct memkind *pmem_kind = NULL;
int err = 0;
errno = 0;
struct stat st;
if (argc > 2) {
fprintf(stderr, "Usage: %s [pmem_kind_dir_path]", argv[0]);
return 1;
} else if (argc == 2) {
if (stat(argv[1], &st) != 0 || !S_ISDIR(st.st_mode)) {
fprintf(stderr, "%s : Invalid path to pmem kind directory", argv[1]);
return 1;
} else {
PMEM_DIR = argv[1];
}
}
//operation below limit the current size of heap
//to show different place of allocation
const struct rlimit heap_limit = { HEAP_LIMIT_SIMULATE, HEAP_LIMIT_SIMULATE };
err = setrlimit(RLIMIT_DATA, &heap_limit);
if (err) {
perror("setrlimit()");
fprintf(stderr, "Unable to set heap limit\n");
return errno ? -errno : 1;
}
char *ptr_default = NULL;
char *ptr_default_not_possible = NULL;
char *ptr_pmem = NULL;
fprintf(stdout,
"This example shows how to allocate memory using standard memory (MEMKIND_DEFAULT) "
"and file-backed kind of memory (PMEM).\nPMEM kind directory: %s\n",
PMEM_DIR);
err = memkind_create_pmem(PMEM_DIR, 0, &pmem_kind);
if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return errno ? -errno : 1;
}
ptr_default = (char *)memkind_malloc(MEMKIND_DEFAULT, size);
if (!ptr_default) {
perror("memkind_malloc()");
fprintf(stderr, "Unable allocate 512 bytes in standard memory");
return errno ? -errno : 1;
}
errno = 0;
ptr_default_not_possible = (char *)memkind_malloc(MEMKIND_DEFAULT,
HEAP_LIMIT_SIMULATE);
if (ptr_default_not_possible) {
perror("memkind_malloc()");
fprintf(stderr,
"Failure, this allocation should not be possible "
"(expected result was NULL), because of setlimit function\n");
return errno ? -errno : 1;
}
if (errno != ENOMEM) {
perror("memkind_malloc()");
fprintf(stderr,
"Failure, this allocation should set errno to ENOMEM value, because of setlimit function\n");
return errno ? -errno : 1;
}
errno = 0;
ptr_pmem = (char *)memkind_malloc(pmem_kind, HEAP_LIMIT_SIMULATE);
if (!ptr_pmem) {
perror("memkind_malloc()");
fprintf(stderr, "Unable allocate HEAP_LIMIT_SIMULATE in file-backed memory");
return errno ? -errno : 1;
}
if (errno != 0) {
perror("memkind_malloc()");
fprintf(stderr, "Failure, this allocation should not set errno value\n");
return errno ? -errno : 1;
}
sprintf(ptr_default, "Hello world from standard memory - ptr_default\n");
sprintf(ptr_pmem, "Hello world from file-backed memory - ptr_pmem\n");
fprintf(stdout, "%s", ptr_default);
fprintf(stdout, "%s", ptr_pmem);
memkind_free(MEMKIND_DEFAULT, ptr_default);
memkind_free(pmem_kind, ptr_pmem);
err = memkind_destroy_kind(pmem_kind);
if (err) {
perror("memkind_destroy_kind()");
fprintf(stderr, "Unable to destroy pmem partition\n");
return errno ? -errno : 1;
}
fprintf(stdout, "Memory was successfully allocated and released.\n");
return 0;
}
+166
View File
@@ -0,0 +1,166 @@
/*
* Copyright (c) 2018 Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include "pmem_allocator.h"
#include <sys/stat.h>
#include <iostream>
#include <vector>
#include <list>
#include <map>
#include <string>
#include <scoped_allocator>
#include <cassert>
#define STL_VECTOR_TEST
#define STL_LIST_TEST
#if _GLIBCXX_USE_CXX11_ABI
#define STL_VEC_STRING_TEST
#define STL_MAP_INT_STRING_TEST
#endif
void cpp_allocator_test(const char *pmem_directory)
{
std::cout << "TEST SCOPE: HELLO" << std::endl;
size_t pmem_max_size = 1024*1024*1024;
#ifdef STL_VECTOR_TEST
{
std::cout << "VECTOR OPEN" << std::endl;
pmem::allocator<int> alc{ pmem_directory, pmem_max_size };
std::vector<int, pmem::allocator<int>> vector{ alc };
for (int i = 0; i < 20; ++i) {
vector.push_back(0xDEAD + i);
assert(vector.back() == 0xDEAD + i);
}
std::cout << "VECTOR CLOSE" << std::endl;
}
#endif
#ifdef STL_LIST_TEST
{
std::cout << "LIST OPEN" << std::endl;
pmem::allocator<int> alc{ pmem_directory, pmem_max_size };
std::list<int, pmem::allocator<int>> list{ alc };
const int nx2 = 4;
for (int i = 0; i < nx2; ++i) {
list.emplace_back(0xBEAC011 + i);
assert(list.back() == 0xBEAC011 + i);
}
for (int i = 0; i < nx2; ++i) {
list.pop_back();
}
std::cout << "LIST CLOSE" << std::endl;
}
#endif
#ifdef STL_VEC_STRING_TEST
{
std::cout << "STRINGED VECTOR OPEN" << std::endl;
typedef pmem::allocator<char> str_alloc_t;
typedef std::basic_string<char, std::char_traits<char>, str_alloc_t>
pmem_string;
typedef pmem::allocator<pmem_string> vec_alloc_t;
vec_alloc_t vec_alloc{ pmem_directory, pmem_max_size };
str_alloc_t str_alloc{ pmem_directory, pmem_max_size };
std::vector<pmem_string, std::scoped_allocator_adaptor<vec_alloc_t> >
vec{ std::scoped_allocator_adaptor<vec_alloc_t>(vec_alloc) };
pmem_string arg{ "Very very loooong striiiing", str_alloc };
vec.push_back(arg);
assert(vec.back() == arg);
std::cout << "STRINGED VECTOR CLOSE" << std::endl;
}
#endif
#ifdef STL_MAP_INT_STRING_TEST
{
std::cout << "INT_STRING MAP OPEN" << std::endl;
typedef std::basic_string<char, std::char_traits<char>, pmem::allocator<char>>
pmem_string;
typedef int key_t;
typedef pmem_string value_t;
typedef pmem::allocator<char> allocator_t;
typedef std::map<key_t, value_t, std::less<key_t>, std::scoped_allocator_adaptor<allocator_t>>
map_t;
allocator_t allocator( pmem_directory, pmem_max_size );
value_t source_str1("Lorem ipsum dolor ", allocator);
value_t source_str2("sit amet consectetuer adipiscing elit", allocator );
map_t target_map{ std::scoped_allocator_adaptor<allocator_t>(allocator) };
target_map[key_t(165)] = source_str1;
assert(target_map[key_t(165)] == source_str1);
target_map[key_t(165)] = source_str2;
assert(target_map[key_t(165)] == source_str2);
std::cout << "INT_STRING MAP CLOSE" << std::endl;
}
#endif
std::cout << "TEST SCOPE: GOODBYE" << std::endl;
}
int main(int argc, char *argv[])
{
const char *pmem_directory = "/tmp/";
if (argc > 2) {
std::cerr << "Usage: pmem_cpp_allocator [directory path]\n"
<< "\t[directory path] - directory where temporary file is created (default = \"/tmp/\")"
<< std::endl;
return 0;
} else if (argc == 2) {
struct stat st;
if (stat(argv[1], &st) != 0 || !S_ISDIR(st.st_mode)) {
fprintf(stderr,"%s : Invalid path to pmem kind directory", argv[1]);
return 1;
}
pmem_directory = argv[1];
}
cpp_allocator_test(pmem_directory);
return 0;
}
+118
View File
@@ -0,0 +1,118 @@
/*
* Copyright (c) 2018 Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdio.h>
#include <errno.h>
#include <sys/stat.h>
static char *PMEM_DIR = "/tmp/";
static const size_t PMEM_PART_SIZE = MEMKIND_PMEM_MIN_SIZE + 4 * 1024;
int main(int argc, char **argv)
{
const size_t size = 512;
struct memkind *pmem_kind = NULL;
struct stat st;
const int arraySize = 100;
char *ptr[100] = { NULL };
int i = 0;
int err = 0;
if (argc > 2) {
fprintf(stderr, "Usage: %s [pmem_kind_dir_path]", argv[0]);
return 1;
} else if (argc == 2) {
if (stat(argv[1], &st) != 0 || !S_ISDIR(st.st_mode)) {
fprintf(stderr, "%s : Invalid path to pmem kind directory", argv[1]);
return 1;
} else {
PMEM_DIR = argv[1];
}
}
fprintf(stdout,
"This example shows how to use memkind_free with unknown kind as a parameter.\n");
err = memkind_create_pmem(PMEM_DIR, PMEM_PART_SIZE, &pmem_kind);
if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return errno ? -errno : 1;
}
for (i = 0; i < arraySize; ++i) {
if (i < 50) {
ptr[i] = memkind_malloc(MEMKIND_DEFAULT, size);
if (ptr[i] == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate memkind default\n");
return errno ? -errno : 1;
}
} else {
ptr[i] = memkind_malloc(pmem_kind, size);
if (ptr[i] == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem\n");
return errno ? -errno : 1;
}
}
}
fprintf(stdout,
"Memory was successfully allocated in default kind and pmem kind.\n");
sprintf(ptr[10], "Hello world from standard memory - ptr[10].\n");
sprintf(ptr[40], "Hello world from standard memory - ptr[40].\n");
sprintf(ptr[80], "Hello world from persistent memory - ptr[80].\n");
fprintf(stdout, "%s", ptr[10]);
fprintf(stdout, "%s", ptr[40]);
fprintf(stdout, "%s", ptr[80]);
fprintf(stdout, "Free memory without specifying kind.\n");
for (i = 0; i < arraySize; ++i) {
memkind_free(NULL, ptr[i]);
}
err = memkind_destroy_kind(pmem_kind);
if (err) {
perror("memkind_destroy_kind()");
fprintf(stderr, "Unable to destroy pmem partition\n");
return errno ? -errno : 1;
}
fprintf(stdout, "Memory was successfully released.\n");
return 0;
}
+126
View File
@@ -0,0 +1,126 @@
/*
* Copyright (c) 2018 Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdio.h>
#include <errno.h>
#include <sys/stat.h>
#define PMEM_MAX_SIZE (1024 * 1024 * 32)
static char *PMEM_DIR = "/tmp/";
int main(int argc, char *argv[])
{
struct memkind *pmem_kinds[10] = {NULL};
struct memkind *pmem_kind = NULL;
struct memkind *pmem_kind_unlimited = NULL;
int err = 0, i = 0;
struct stat st;
if (argc > 2) {
fprintf(stderr, "Usage: %s [pmem_kind_dir_path]", argv[0]);
return 1;
} else if (argc == 2) {
if (stat(argv[1], &st) != 0 || !S_ISDIR(st.st_mode)) {
fprintf(stderr, "%s : Invalid path to pmem kind directory", argv[1]);
return 1;
} else {
PMEM_DIR = argv[1];
}
}
fprintf(stdout,
"This example shows how to create and destroy pmem kind with defined or unlimited size."
"\nPMEM kind directory: %s\n",
PMEM_DIR);
/* Create PMEM partition with specific size */
err = memkind_create_pmem(PMEM_DIR, PMEM_MAX_SIZE, &pmem_kind);
if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return errno ? -errno : 1;
}
/* Create PMEM partition with unlimited size */
err = memkind_create_pmem(PMEM_DIR, 0, &pmem_kind_unlimited);
if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return errno ? -errno : 1;
}
/* and delete them */
err = memkind_destroy_kind(pmem_kind);
if (err) {
perror("memkind_destroy_kind()");
fprintf(stderr, "Unable to destroy pmem partition\n");
return errno ? -errno : 1;
}
err = memkind_destroy_kind(pmem_kind_unlimited);
if (err) {
perror("memkind_destroy_kind()");
fprintf(stderr, "Unable to destroy pmem partition\n");
return errno ? -errno : 1;
}
/* Create many PMEM kinds */
for (i = 0; i < 10; i++) {
err = memkind_create_pmem(PMEM_DIR, PMEM_MAX_SIZE, &pmem_kinds[i]);
if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return errno ? -errno : 1;
}
}
/* and delete them */
for (i = 0; i < 10; i++) {
err = memkind_destroy_kind(pmem_kinds[i]);
if (err) {
perror("memkind_pmem_destroy()");
fprintf(stderr, "Unable to destroy pmem partition\n");
return errno ? -errno : 1;
}
}
fprintf(stdout, "PMEM kinds have been successfully created and destroyed.");
return 0;
}
+135
View File
@@ -0,0 +1,135 @@
/*
* Copyright (c) 2015 - 2018 Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdio.h>
#include <errno.h>
#include <sys/stat.h>
#define PMEM_MAX_SIZE (1024 * 1024 * 32)
static char *PMEM_DIR = "/tmp/";
int main(int argc, char *argv[])
{
struct memkind *pmem_kind = NULL;
int err = 0;
struct stat st;
if (argc > 2) {
fprintf(stderr, "Usage: %s [pmem_kind_dir_path]", argv[0]);
return 1;
} else if (argc == 2) {
if (stat(argv[1], &st) != 0 || !S_ISDIR(st.st_mode)) {
fprintf(stderr, "%s : Invalid path to pmem kind directory", argv[1]);
return 1;
} else {
PMEM_DIR = argv[1];
}
}
fprintf(stdout,
"This example shows how to allocate memory and possibility to exceed pmem kind size."
"\nPMEM kind directory: %s\n",
PMEM_DIR);
/* Create PMEM partition with specific size */
err = memkind_create_pmem(PMEM_DIR, PMEM_MAX_SIZE, &pmem_kind);
if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return errno ? -errno : 1;
}
char *pmem_str1 = NULL;
char *pmem_str2 = NULL;
char *pmem_str3 = NULL;
char *pmem_str4 = NULL;
// allocate 512 Bytes of 32 MB available
pmem_str1 = (char *)memkind_malloc(pmem_kind, 512);
if (pmem_str1 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str1)\n");
return errno ? -errno : 1;
}
// allocate 8 MB of 31.9 MB available
pmem_str2 = (char *)memkind_malloc(pmem_kind, 8 * 1024 * 1024);
if (pmem_str2 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str11)\n");
return errno ? -errno : 1;
}
// allocate 16 MB of 23.9 MB available
pmem_str3 = (char *)memkind_malloc(pmem_kind, 16 * 1024 * 1024);
if (pmem_str3 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str12)\n");
return errno ? -errno : 1;
}
// allocate 16 MB of 7.9 MB available -- Out Of Memory expected
pmem_str4 = (char *)memkind_malloc(pmem_kind, 16 * 1024 * 1024);
if (pmem_str4 != NULL) {
perror("memkind_malloc()");
fprintf(stderr,
"Failure, this allocation should not be possible (expected result was NULL)\n");
return errno ? -errno : 1;
}
sprintf(pmem_str1, "Hello world from pmem - pmem_str1\n");
sprintf(pmem_str2, "Hello world from pmem - pmem_str2\n");
sprintf(pmem_str3, "Hello world from persistent memory - pmem_str3\n");
fprintf(stdout, "%s", pmem_str1);
fprintf(stdout, "%s", pmem_str2);
fprintf(stdout, "%s", pmem_str3);
memkind_free(pmem_kind, pmem_str1);
memkind_free(pmem_kind, pmem_str2);
memkind_free(pmem_kind, pmem_str3);
err = memkind_destroy_kind(pmem_kind);
if (err) {
perror("memkind_destroy_kind()");
fprintf(stderr, "Unable to destroy pmem partition\n");
return errno ? -errno : 1;
}
fprintf(stdout, "Memory was successfully allocated and released.\n");
return 0;
}
+105
View File
@@ -0,0 +1,105 @@
/*
* Copyright (c) 2018 Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdio.h>
#include <errno.h>
#include <sys/stat.h>
static char *PMEM_DIR = "/tmp/";
int main(int argc, char *argv[])
{
struct memkind *pmem_kind_unlimited = NULL;
int err = 0;
struct stat st;
if (argc > 2) {
fprintf(stderr, "Usage: %s [pmem_kind_dir_path]", argv[0]);
return 1;
} else if (argc == 2) {
if (stat(argv[1], &st) != 0 || !S_ISDIR(st.st_mode)) {
fprintf(stderr, "%s : Invalid path to pmem kind directory ", argv[1]);
return 1;
} else {
PMEM_DIR = argv[1];
}
}
fprintf(stdout,
"This example shows how to allocate memory with unlimited kind size."
"nPMEM kind directory: %s\n",
PMEM_DIR);
/* Create PMEM partition with unlimited size */
err = memkind_create_pmem(PMEM_DIR, 0, &pmem_kind_unlimited);
if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return errno ? -errno : 1;
}
char *pmem_str10 = NULL;
char *pmem_str11 = NULL;
/* Huge allocation */
pmem_str10 = (char *)memkind_malloc(pmem_kind_unlimited, 32 * 1024 * 1024);
if (pmem_str10 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str10)\n");
return errno ? -errno : 1;
}
/* Another huge allocation, kind size is only limited by OS resources */
pmem_str11 = (char *)memkind_malloc(pmem_kind_unlimited, 32 * 1024 * 1024);
if (pmem_str11 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str11)\n");
return errno ? -errno : 1;
}
memkind_free(pmem_kind_unlimited, pmem_str10);
memkind_free(pmem_kind_unlimited, pmem_str11);
err = memkind_destroy_kind(pmem_kind_unlimited);
if (err) {
perror("memkind_destroy_kind()");
fprintf(stderr, "Unable to destroy pmem partition\n");
return errno ? -errno : 1;
}
fprintf(stdout, "Memory was successfully allocated and released.");
return 0;
}
+138
View File
@@ -0,0 +1,138 @@
/*
* Copyright (c) 2018 Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <pthread.h>
#include <sys/stat.h>
#include <unistd.h>
#define PMEM_MAX_SIZE (1024 * 1024 * 32)
#define NUM_THREADS 10
static char *PMEM_DIR = "/tmp/";
void *thread_ind(void *arg);
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
int main(int argc, char *argv[])
{
int err = 0;
struct stat st;
if (argc > 2) {
fprintf(stderr, "Usage: %s [pmem_kind_dir_path]", argv[0]);
return 1;
} else if (argc == 2) {
if (stat(argv[1], &st) != 0 || !S_ISDIR(st.st_mode)) {
fprintf(stderr, "%s : Invalid path to pmem kind directory", argv[1]);
return 1;
} else {
PMEM_DIR = argv[1];
}
}
fprintf(stdout,
"This example shows how to use multithreading with independent pmem kinds."
"\nPMEM kind directory: %s\n",
PMEM_DIR);
pthread_t pmem_threads[NUM_THREADS];
int t;
/* Lets create many independent threads */
for (t = 0; t < NUM_THREADS; t++) {
err = pthread_create(&pmem_threads[t], NULL, thread_ind, NULL);
if (err) {
fprintf(stderr, "Unable to create a thread\n");
return 1;
}
}
sleep(1);
pthread_cond_broadcast(&cond);
for (t = 0; t < NUM_THREADS; t++) {
err = pthread_join(pmem_threads[t], NULL);
if (err) {
fprintf(stderr, "Thread join failed\n");
return 1;
}
}
fprintf(stdout, "Threads successfully allocated memory in the PMEM kinds.");
return 0;
}
void *thread_ind(void *arg)
{
struct memkind *pmem_kind;
pthread_mutex_lock(&mutex);
pthread_cond_wait(&cond, &mutex);
pthread_mutex_unlock(&mutex);
/* Create a pmem kind in thread */
int err = memkind_create_pmem(PMEM_DIR, PMEM_MAX_SIZE, &pmem_kind);
if (err) {
perror("thread memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return NULL;
}
/* Alloc something */
void *test = memkind_malloc(pmem_kind, 32);
if (test == NULL) {
perror("thread memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem (test)\n");
return NULL;
}
/* Free resources */
memkind_free(pmem_kind, test);
/* And destroy pmem kind */
err = memkind_destroy_kind(pmem_kind);
if (err) {
perror("thread memkind_pmem_destroy()");
fprintf(stderr, "Unable to destroy pmem partition\n");
}
return NULL;
}
+159
View File
@@ -0,0 +1,159 @@
/*
* Copyright (c) 2018 Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <pthread.h>
#include <sys/stat.h>
#include <unistd.h>
#define NUM_THREADS 10
static char *PMEM_DIR = "/tmp/";
struct arg_struct {
int id;
struct memkind *kind;
int **ptr;
};
void *thread_onekind(void *arg);
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
int main(int argc, char *argv[])
{
struct memkind *pmem_kind_unlimited = NULL;
int err = 0;
struct stat st;
if (argc > 2) {
fprintf(stderr, "Usage: %s [pmem_kind_dir_path]", argv[0]);
return 1;
} else if (argc == 2) {
if (stat(argv[1], &st) != 0 || !S_ISDIR(st.st_mode)) {
fprintf(stderr, "%s : Invalid path to pmem kind directory", argv[1]);
return 1;
} else {
PMEM_DIR = argv[1];
}
}
fprintf(stdout,
"This example shows how to use multithreading with one main pmem kind."
"\nPMEM kind directory: %s\n",
PMEM_DIR);
/* Create PMEM partition with unlimited size */
err = memkind_create_pmem(PMEM_DIR, 0, &pmem_kind_unlimited);
if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return errno ? -errno : 1;
}
/* Create a few threads which will access to our main pmem_kind */
pthread_t pmem_threads[NUM_THREADS];
int *pmem_tint[NUM_THREADS][100];
int t = 0, i = 0;
struct arg_struct *args[NUM_THREADS];
for (t = 0; t<NUM_THREADS; t++) {
args[t] = malloc(sizeof(struct arg_struct));
args[t]->id = t;
args[t]->ptr = &pmem_tint[t][0];
args[t]->kind = pmem_kind_unlimited;
err = pthread_create(&pmem_threads[t], NULL, thread_onekind, (void *)args[t]);
if (err) {
fprintf(stderr, "Unable to create a thread\n");
return 1;
}
}
sleep(1);
pthread_cond_broadcast(&cond);
for (t = 0; t < NUM_THREADS; t++) {
err = pthread_join(pmem_threads[t], NULL);
if (err) {
fprintf(stderr, "Thread join failed\n");
return 1;
}
}
/* Check if we can read the values outside of threads and free resources */
for (t = 0; t < NUM_THREADS; t++) {
for (i = 0; i < 100; i++) {
if(*pmem_tint[t][i] != t) {
perror("read thread memkind_malloc()");
fprintf(stderr, "pmem_tint value has not been saved correctly in the thread\n");
return 1;
}
memkind_free(args[t]->kind, *(args[t]->ptr+i));
}
free(args[t]);
}
fprintf(stdout, "Threads successfully allocated memory in the PMEM kind.");
return 0;
}
void *thread_onekind(void *arg)
{
struct arg_struct *args = (struct arg_struct *)arg;
int i;
pthread_mutex_lock(&mutex);
pthread_cond_wait(&cond, &mutex);
pthread_mutex_unlock(&mutex);
/* Lets alloc int and put there thread ID */
for (i = 0; i < 100; i++) {
*(args->ptr+i) = (int *)memkind_malloc(args->kind, sizeof(int));
if (*(args->ptr+i) == NULL) {
perror("thread memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem int\n");
return NULL;
}
**(args->ptr+i) = args->id;
}
return NULL;
}
+172
View File
@@ -0,0 +1,172 @@
/*
* Copyright (c) 2018 Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY LOG OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <memkind.h>
#include <stdio.h>
#include <errno.h>
#include <sys/stat.h>
static char *PMEM_DIR = "/tmp/";
int main(int argc, char *argv[])
{
struct memkind *pmem_kind_unlimited = NULL;
int err = 0;
struct stat st;
if (argc > 2) {
fprintf(stderr, "Usage: %s [pmem_kind_dir_path]", argv[0]);
return 1;
} else if (argc == 2) {
if (stat(argv[1], &st) != 0 || !S_ISDIR(st.st_mode)) {
fprintf(stderr, "%s : Invalid path to pmem kind directory", argv[1]);
return 1;
} else {
PMEM_DIR = argv[1];
}
}
fprintf(stdout,
"This example shows difference between the expected and the actual allocation size."
"\nPMEM kind directory: %s\n",
PMEM_DIR);
/* Create PMEM partition with unlimited size */
err = memkind_create_pmem(PMEM_DIR, 0, &pmem_kind_unlimited);
if (err) {
perror("memkind_create_pmem()");
fprintf(stderr, "Unable to create pmem partition err=%d errno=%d\n", err,
errno);
return errno ? -errno : 1;
}
char *pmem_str10 = NULL;
char *pmem_str11 = NULL;
char *pmem_str12 = NULL;
/* 32 bytes allocation */
pmem_str10 = (char *)memkind_malloc(pmem_kind_unlimited, 32);
if (pmem_str10 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str10)\n");
return errno ? -errno : 1;
}
/* Check real usable size for this allocation */
if (memkind_malloc_usable_size(pmem_kind_unlimited, pmem_str10) != 32) {
perror("memkind_default_malloc_usable_size()");
fprintf(stderr, "Wrong usable size\n");
return 1;
}
/* 31 bytes allocation */
pmem_str11 = (char *)memkind_malloc(pmem_kind_unlimited, 31);
if (pmem_str11 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str11)\n");
return errno ? -errno : 1;
}
/* Check real usable size for this allocation, its 32 again */
if (memkind_malloc_usable_size(pmem_kind_unlimited, pmem_str11) != 32) {
perror("memkind_default_malloc_usable_size()");
fprintf(stderr, "Wrong usable size\n");
return 1;
}
/* 33 bytes allocation */
pmem_str12 = (char *)memkind_malloc(pmem_kind_unlimited, 33);
if (pmem_str11 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str12)\n");
return errno ? -errno : 1;
}
/* Check real usable size for this allocation, its 48 now */
if (memkind_malloc_usable_size(pmem_kind_unlimited, pmem_str12) != 48) {
perror("memkind_default_malloc_usable_size()");
fprintf(stderr, "Wrong usable size\n");
return 1;
}
memkind_free(pmem_kind_unlimited, pmem_str10);
memkind_free(pmem_kind_unlimited, pmem_str11);
memkind_free(pmem_kind_unlimited, pmem_str12);
/* 5MB allocation */
pmem_str10 = (char *)memkind_malloc(pmem_kind_unlimited, 5 * 1024 * 1024);
if (pmem_str10 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str10)\n");
return errno ? -errno : 1;
}
/* Check real usable size for this allocation */
if (memkind_malloc_usable_size(pmem_kind_unlimited,
pmem_str10) != 5 * 1024 * 1024) {
perror("memkind_default_malloc_usable_size()");
fprintf(stderr, "Wrong usable size\n");
return 1;
}
/* 5MB + 1B allocation */
pmem_str11 = (char *)memkind_malloc(pmem_kind_unlimited, 5 * 1024 * 1024 + 1);
if (pmem_str11 == NULL) {
perror("memkind_malloc()");
fprintf(stderr, "Unable to allocate pmem string (pmem_str11)\n");
return errno ? -errno : 1;
}
/* Check real usable size for this allocation, its 6MB now */
if (memkind_malloc_usable_size(pmem_kind_unlimited,
pmem_str11) != 6 * 1024 * 1024) {
perror("memkind_default_malloc_usable_size()");
fprintf(stderr, "Wrong usable size\n");
return 1;
}
memkind_free(pmem_kind_unlimited, pmem_str10);
memkind_free(pmem_kind_unlimited, pmem_str11);
err = memkind_destroy_kind(pmem_kind_unlimited);
if (err) {
perror("memkind_destroy_kind()");
fprintf(stderr, "Unable to destroy pmem partition\n");
return errno ? -errno : 1;
}
fprintf(stdout,
"The real size of the allocation has been successfully read.");
return 0;
}
+134
View File
@@ -0,0 +1,134 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#include <hbwmalloc.h>
#include <stddef.h>
#include <new>
/*
* Header file for the C++ allocator compatible with the C++ standard library allocator concepts.
* More details in hbwallocator(3) man page.
* Note: memory heap management is based on hbwmalloc, refer to the hbwmalloc man page for more information.
*
* Functionality defined in this header is considered as EXPERIMENTAL API.
* API standards are described in memkind(3) man page.
*/
namespace hbw
{
template <class T>
class allocator
{
public:
/*
* Public member types required and defined by the standard library allocator concepts.
*/
typedef size_t size_type;
typedef ptrdiff_t difference_type;
typedef T *pointer;
typedef const T *const_pointer;
typedef T &reference;
typedef const T &const_reference;
typedef T value_type;
template <class U>
struct rebind {
typedef hbw::allocator<U> other;
};
/*
* Public member functions required and defined by the standard library allocator concepts.
*/
allocator() throw() { }
template <class U>
allocator(const allocator<U> &) throw() { }
~allocator() throw() { }
pointer address(reference x) const
{
return &x;
}
const_pointer address(const_reference x) const
{
return &x;
}
/*
* Allocates n*sizeof(T) bytes of high bandwidth memory using hbw_malloc().
* Throws std::bad_alloc when cannot allocate memory.
*/
pointer allocate(size_type n, const void * = 0)
{
if (n > this->max_size()) {
throw std::bad_alloc();
}
pointer result = static_cast<pointer>(hbw_malloc(n * sizeof(T)));
if (!result) {
throw std::bad_alloc();
}
return result;
}
/*
* Deallocates memory associated with pointer returned by allocate() using hbw_free().
*/
void deallocate(pointer p, size_type n)
{
hbw_free(static_cast<void *>(p));
}
size_type max_size() const throw()
{
return size_t(-1) / sizeof(T);
}
void construct(pointer p, const_reference val)
{
::new(p) value_type(val);
}
void destroy(pointer p)
{
p->~T();
}
};
template <class T, class U>
bool operator==(const allocator<T> &, const allocator<U> &)
{
return true;
}
template <class T, class U>
bool operator!=(const allocator<T> &, const allocator<U> &)
{
return false;
}
}
+238
View File
@@ -0,0 +1,238 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#include <stdlib.h>
/*
* Header file for the high bandwidth memory interface.
*
* This file defines the external API's and enumerations for the
* hbwmalloc library. These interfaces define a heap manager that
* targets the high bandwidth memory numa nodes.
*
* hbwmalloc.h functionality is considered as stable API (STANDARD API).
*
* Please read hbwmalloc(3) man page for or more details.
*/
/*
* Fallback policy.
*
* Policy that determines behavior when there is not enough free high
* bandwidth memory to satisfy a user request. This enum is used with
* hbw_get_policy() and hbw_set_policy().
*/
typedef enum {
/*
* If insufficient high bandwidth memory pages on nearest NUMA node are
* available then OOM killer will be triggered.
*/
HBW_POLICY_BIND = 1,
/*
* If insufficient high bandwidth memory pages are available fall
* back on standard memory pages.
*/
HBW_POLICY_PREFERRED = 2,
/*
* Interleave pages across high bandwidth nodes. If insufficient memory
* pages are available then OOM killer will be triggered.
*/
HBW_POLICY_INTERLEAVE = 3,
/*
* If insufficient high bandwidth memory pages are available then
* OOM killer will be triggered.
*/
HBW_POLICY_BIND_ALL = 4,
} hbw_policy_t;
/*
* Page size selection.
*
* The hbw_posix_memalign_psize() API gives the user the option to
* select the page size from this enumerated list.
*/
typedef enum {
/*
* The four kilobyte page size option. Note that with transparent huge
* pages enabled these allocations may be promoted by the operating system
* to two megabyte pages.
*/
HBW_PAGESIZE_4KB = 1,
/*
* The two megabyte page size option.
*/
HBW_PAGESIZE_2MB = 2,
/*
* This option is deprecated.
* Allocate high bandwidth memory using 1GB chunks backed by huge pages.
*/
HBW_PAGESIZE_1GB_STRICT = 3,
/*
* This option is deprecated.
* Allocate high bandwidth memory using 1GB chunks backed by huge pages.
*/
HBW_PAGESIZE_1GB = 4,
/*
* Helper representing value of the last enum element incremented by 1.
* Shall not be treated as a valid value for functions taking hbw_pagesize_t
* as parameter.
*/
HBW_PAGESIZE_MAX_VALUE
} hbw_pagesize_t;
/*
* Flags for hbw_verify_ptr function
*/
enum {
/*
* This option touches first byte of all pages in address range starting from "addr" to "addr" + "size"
* by read and write (so the content will be overwitten by the same data as it was read).
*/
HBW_TOUCH_PAGES = (1 << 0)
};
/*
* Returns the current fallback policy when insufficient high bandwidth memory
* is available.
*/
hbw_policy_t hbw_get_policy(void);
/*
* Sets the current fallback policy. The policy can be modified only once in
* the lifetime of an application and before calling hbw_*alloc() or
* hbw_posix_memalign*() function.
* Note: If the policy is not set, than HBW_POLICY_PREFERRED will be used by
* default.
*
* Returns:
* 0: on success
* EPERM: if hbw_set_policy () was called more than once
* EINVAL: if mode argument was neither HBW_POLICY_PREFERRED, HBW_POLICY_BIND, HBW_POLICY_BIND_ALL nor HBW_POLICY_INTERLEAVE
*/
int hbw_set_policy(hbw_policy_t mode);
/*
* Verifies high bandwidth memory availability.
* Returns:
* 0: if high bandwidth memory is available
* ENODEV: if high-bandwidth memory is unavailable.
*/
int hbw_check_available(void);
/*
* Verifies if allocated memory fully fall into high bandwidth memory.
* Returns:
* 0: if memory in address range from "addr" to "addr" + "size" is allocated in high bandwidth memory
* -1: if any region of memory was not allocated in high bandwidth memory
* EINVAL: if addr is NULL, size equals 0 or flags contained unsupported bit set
* EFAULT: could not verify memory
*/
int hbw_verify_memory_region(void *addr, size_t size, int flags);
/*
* Allocates size bytes of uninitialized high bandwidth memory.
* The allocated space is suitably aligned (after possible pointer
* coercion) for storage of any type of object. If size is zero then
* hbw_malloc() returns NULL.
*/
void *hbw_malloc(size_t size);
/*
* Allocates space for num objects in high bandwidth memory, each size bytes
* in length.
* The result is identical to calling hbw_malloc() with an argument of
* num*size, with the exception that the allocated memory is explicitly
* initialized to zero bytes.
* If num or size is 0, then hbw_calloc() returns NULL.
*/
void *hbw_calloc(size_t num, size_t size);
/*
* Allocates size bytes of high bandwidth memory such that the allocation's
* base address is an even multiple of alignment, and returns the allocation
* in the value pointed to by memptr. The requested alignment must be a power
* of 2 at least as large as sizeof(void *).
* Returns:
* 0: on success
* ENOMEM: if there was insufficient memory to satisfy the request
* EINVAL: if the alignment parameter was not a power of two, or was less than sizeof(void *)
*/
int hbw_posix_memalign(void **memptr, size_t alignment, size_t size);
/*
* Allocates size bytes of high bandwidth memory such that the allocation's
* base address is an even multiple of alignment, and returns the allocation
* in the value pointed to by memptr. The requested alignment must be a power
* of 2 at least as large as sizeof(void *). The memory will be allocated
* using pages determined by the pagesize variable.
* Returns:
* 0: on success
* ENOMEM: if there was insufficient memory to satisfy the request
* EINVAL: if the alignment parameter was not a power of two, or was less than sizeof(void *)
*/
int hbw_posix_memalign_psize(void **memptr, size_t alignment, size_t size,
hbw_pagesize_t pagesize);
/*
* Changes the size of the previously allocated memory referenced by ptr to
* size bytes of the specified kind. The contents of the memory are unchanged
* up to the lesser of the new and old size.
* If the new size is larger, the contents of the newly allocated portion
* of the memory are undefined.
* Upon success, the memory referenced by ptr is freed and a pointer to the
* newly allocated high bandwidth memory is returned.
* Note: memkind_realloc() may move the memory allocation, resulting in a
* different return value than ptr.
* If ptr is NULL, the hbw_realloc() function behaves identically to
* hbw_malloc() for the specified size. The address ptr, if not NULL,
* was returned by a previous call to hbw_malloc(), hbw_calloc(),
* hbw_realloc(), or hbw_posix_memalign(). Otherwise, or if hbw_free(ptr)
* was called before, undefined behavior occurs.
* Note: hbw_realloc() cannot be used with a pointer returned by
* hbw_posix_memalign_psize().
*/
void *hbw_realloc(void *ptr, size_t size);
/*
* Causes the allocated memory referenced by ptr to be made
* available for future allocations. If ptr is NULL, no action occurs.
* The address ptr, if not NULL, must have been returned by a previous call
* to hbw_malloc(), hbw_calloc(), hbw_realloc(), hbw_posix_memalign(), or
* hbw_posix_memalign_psize(). Otherwise, if hbw_free(ptr) was called before,
* undefined behavior occurs.
*/
void hbw_free(void *ptr);
#ifdef __cplusplus
}
#endif
+325
View File
@@ -0,0 +1,325 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#include <sys/types.h>
/**
* Header file for the memkind heap manager.
* More details in memkind(3) man page.
*
* API standards are described in memkind(3) man page.
*/
/// \warning EXPERIMENTAL API
#define _MEMKIND_BIT(N) (1ull << N)
/// \brief Memkind memory types
/// \warning EXPERIMENTAL API
typedef enum memkind_memtype_t {
/**
* Select standard memory, the same as process use.
*/
MEMKIND_MEMTYPE_DEFAULT = _MEMKIND_BIT(0),
/**
* Select high bandwidth memory (HBM).
* There must be at least two memories with different bandwidth to
* determine the HBM.
*/
MEMKIND_MEMTYPE_HIGH_BANDWIDTH = _MEMKIND_BIT(1)
} memkind_memtype_t;
#undef _MEMKIND_BIT
/// \brief Memkind policy
/// \warning EXPERIMENTAL API
typedef enum memkind_policy_t {
/**
* Allocate local memory.
* If there is not enough memory to satisfy the request errno is set to
* ENOMEM and the allocated pointer is set to NULL.
*/
MEMKIND_POLICY_BIND_LOCAL = 0,
/**
* Memory locality is ignored.
* If there is not enough memory to satisfy the request errno is set to
* ENOMEM and the allocated pointer is set to NULL.
*/
MEMKIND_POLICY_BIND_ALL,
/**
* Allocate preferred memory that is local.
* If there is not enough preferred memory to satisfy the request or
* preferred memory is not available, the allocation will fall back on any
* other memory.
*/
MEMKIND_POLICY_PREFERRED_LOCAL,
/**
* Interleave allocation across local memory.
* For n memory types the allocation will be interleaved across all of
* them.
*/
MEMKIND_POLICY_INTERLEAVE_LOCAL,
/**
* Interleave allocation. Locality is ignored.
* For n memory types the allocation will be interleaved across all of
* them.
*/
MEMKIND_POLICY_INTERLEAVE_ALL,
/**
* Max policy value.
*/
MEMKIND_POLICY_MAX_VALUE
} memkind_policy_t;
/// \brief Memkind bits definition
/// \warning EXPERIMENTAL API
/// \note The bits specify flags and masks. Bits <0,1,2,...,7> are reserved for page size, where page sizes are encoded
/// by base-2 logarithm. If the page size bits are set to zero value, than default page size will be used.
typedef enum memkind_bits_t {
MEMKIND_MASK_PAGE_SIZE_2MB = 21ull, /**< Allocations backed by 2 MB page size (2^21 = 2MB) */
} memkind_bits_t;
/// \brief Memkind type definition
/// \warning EXPERIMENTAL API
typedef struct memkind *memkind_t;
/// \brief Memkind constant values
/// \warning EXPERIMENTAL API
enum memkind_const {
MEMKIND_MAX_KIND = 512, /**< Maximum number of kinds */
MEMKIND_ERROR_MESSAGE_SIZE = 128, /**< Error message size */
MEMKIND_PMEM_MIN_SIZE = (1024 * 1024 * 16) /**< The minimum size which allows to limit the file-backed memory partition */
};
/// \brief Memkind operation statuses
/// \warning EXPERIMENTAL API
enum {
MEMKIND_SUCCESS = 0, /**< Operation success */
MEMKIND_ERROR_UNAVAILABLE = -1, /**< Error: Memory kind is not available */
MEMKIND_ERROR_MBIND = -2, /**< Error: Call to mbind() failed */
MEMKIND_ERROR_MMAP = -3, /**< Error: Call to mmap() failed */
MEMKIND_ERROR_MALLOC = -6, /**< Error: Call to malloc() failed */
MEMKIND_ERROR_ENVIRON = -12, /**< Error: Unable to parse environment variable */
MEMKIND_ERROR_INVALID = -13, /**< Error: Invalid argument */
MEMKIND_ERROR_TOOMANY = -15, /**< Error: Attempt to initialize more than MEMKIND_MAX_KIND number of kinds */
MEMKIND_ERROR_BADOPS = -17, /**< Error: Invalid memkind_ops structure */
MEMKIND_ERROR_HUGETLB = -18, /**< Error: Unable to allocate huge pages */
MEMKIND_ERROR_MEMTYPE_NOT_AVAILABLE = -20, /**< Error: Requested memory type is not available */
MEMKIND_ERROR_OPERATION_FAILED = -21, /**< Error: Operation failed */
MEMKIND_ERROR_ARENAS_CREATE = -22, /**< Error: Call to jemalloc's arenas.create failed */
MEMKIND_ERROR_RUNTIME = -255 /**< Error: Unspecified run-time error */
};
///
/// \brief Create kind that allocates memory with specific memory type, memory binding policy and flags.
/// \warning EXPERIMENTAL API
/// \note Currently implemented memory type and policy configurations:
///. {MEMKIND_MEMTYPE_DEFAULT, MEMKIND_POLICY_PREFERRED_LOCAL},
///. {MEMKIND_MEMTYPE_HIGH_BANDWIDTH, MEMKIND_POLICY_BIND_LOCAL},
/// {MEMKIND_MEMTYPE_HIGH_BANDWIDTH, MEMKIND_POLICY_PREFERRED_LOCAL},
/// {MEMKIND_MEMTYPE_HIGH_BANDWIDTH, MEMKIND_POLICY_INTERLEAVE_ALL},
/// {MEMKIND_MEMTYPE_DEFAULT | MEMKIND_MEMTYPE_HIGH_BANDWIDTH, MEMKIND_POLICY_INTERLEAVE_ALL}.
/// \param memtype_flags determine the memory types to allocate from by combination of memkind_memtype_t values.
/// This field cannot have zero value.
/// \param policy specify policy for page binding to memory types selected by memtype_flags.
/// This field must be set to memkind_policy_t value. If policy is set to MEMKIND_POLICY_PREFERRED_LOCAL then only one memory
/// type must be selected. Note: the value cannot be set to MEMKIND_POLICY_MAX_VALUE.
/// \param flags the field must be set to a combination of memkind_bits_t values.
/// \param kind pointer to kind which will be created
/// \return Memkind operation status, MEMKIND_SUCCESS on success, MEMKIND_ERROR_MEMTYPE_NOT_AVAILABLE or MEMKIND_ERROR_INVALID on failure
///
int memkind_create_kind(memkind_memtype_t memtype_flags,
memkind_policy_t policy,
memkind_bits_t flags,
memkind_t *kind);
///
/// \brief Destroy previously created kind object, which must have been returned
/// by a call to memkind_create_kind() or memkind_create_pmem().
/// The function has undefined behavior when the handle is invalid or
/// memkind_destroy_kind(kind) was already called before
/// \warning EXPERIMENTAL API
/// \note if the kind was returned by memkind_create_kind() all allocated memory must be freed
/// before kind is destroyed, otherwise this will cause memory leak.
/// \param kind specified memory kind
/// \return Memkind operation status, MEMKIND_SUCCESS on success, MEMKIND_ERROR_OPERATION_FAILED on failure
///
int memkind_destroy_kind(memkind_t kind);
#include "memkind_deprecated.h"
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_REGULAR;
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_DEFAULT;
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_HUGETLB;
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_HBW;
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_HBW_ALL;
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_HBW_PREFERRED;
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_HBW_HUGETLB;
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_HBW_ALL_HUGETLB;
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_HBW_PREFERRED_HUGETLB;
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_HBW_INTERLEAVE;
/// \warning EXPERIMENTAL API
extern memkind_t MEMKIND_INTERLEAVE;
///
/// \brief Get Memkind API version
/// \note STANDARD API
/// \return Version number represented by a single integer number(major * 1000000 + minor * 1000 + patch)
///
int memkind_get_version();
///
/// \brief Convert error number into an error message
/// \note STANDARD API
/// \param err error number
/// \param msg error message
/// \param size size of message
///
void memkind_error_message(int err, char *msg, size_t size);
///
/// \brief Create a new PMEM (file-backed) kind of given size on top of a temporary file
/// in the given directory dir
/// \note STANDARD API
/// \param dir path to specified directory to temporary file
/// \param max_size size limit for kind
/// \param kind pointer to kind which will be created
/// \return Memkind operation status, MEMKIND_SUCCESS on success, other values on failure
///
int memkind_create_pmem(const char *dir, size_t max_size, memkind_t *kind);
///
/// \brief Check if kind is available
/// \note STANDARD API
/// \param kind specified memory kind
/// \return Memkind operation status, MEMKIND_SUCCESS on success, other values on failure
///
int memkind_check_available(memkind_t kind);
/* HEAP MANAGEMENT INTERFACE */
///
/// \brief Allocates size bytes of uninitialized storage of the specified kind
/// \note STANDARD API
/// \param kind specified memory kind
/// \param size number of bytes to allocate
/// \return Pointer to the allocated memory
///
void *memkind_malloc(memkind_t kind, size_t size);
///
/// \brief Obtain size of block of memory allocated with the memkind API
/// \note STANDARD API
/// \param kind specified memory kind
/// \param ptr pointer to the allocated memory
/// \return Number of usable bytes
///
size_t memkind_malloc_usable_size(memkind_t kind, void *ptr);
///
/// \brief Allocates memory of the specified kind for an array of num elements
/// of size bytes each and initializes all bytes in the allocated storage to zero
/// \note STANDARD API
/// \param kind specified memory kind
/// \param num number of objects
/// \param size specified size of each element
/// \return Pointer to the allocated memory
///
void *memkind_calloc(memkind_t kind, size_t num, size_t size);
/// \brief Allocates size bytes of the specified kind and places the address of the allocated memory
/// in *memptr. The address of the allocated memory will be a multiple of alignment,
/// which must be a power of two and a multiple of sizeof(void *)
/// \note EXPERIMENTAL API
/// \param kind specified memory kind
/// \param memptr address of the allocated memory
/// \param alignment specified alignment of bytes
/// \param size specified size of bytes
/// \return Memkind operation status, MEMKIND_SUCCESS on success, EINVAL or ENOMEM on failure
///
int memkind_posix_memalign(memkind_t kind, void **memptr, size_t alignment,
size_t size);
///
/// \brief Reallocates memory of the specified kind
/// \note STANDARD API
/// \param kind specified memory kind
/// \param ptr pointer to the memory block to be reallocated
/// \param size new size for the memory block in bytes
/// \return Pointer to the allocated memory
///
void *memkind_realloc(memkind_t kind, void *ptr, size_t size);
///
/// \brief Free the memory space of the specified kind pointed by ptr
/// \note STANDARD API
/// \param kind specified memory kind
/// \param ptr pointer to the allocated memory
///
void memkind_free(memkind_t kind, void *ptr);
int memkind_fd(struct memkind *kind);
void memkind_pmem_remapfd(struct memkind *kind, int fdNew);
int memkind_tmpfile(const char *dir, int *fd);
memkind_t memkind_get_kind(void *ptr);
#ifdef __cplusplus
}
#endif
@@ -0,0 +1,30 @@
/*
* Copyright (C) 2017 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#include <memkind.h>
void heap_manager_init(struct memkind *kind);
void heap_manager_free(struct memkind *kind, void *ptr);
@@ -0,0 +1,68 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#ifndef MEMKIND_INTERNAL_API
#warning "DO NOT INCLUDE THIS FILE! IT IS INTERNAL MEMKIND API AND SOON WILL BE REMOVED FROM BIN & DEVEL PACKAGES"
#endif
#include <memkind.h>
#include <jemalloc/jemalloc.h>
#include <memkind/internal/memkind_private.h>
/*
* Header file for the jemalloc arena allocation memkind operations.
* More details in memkind_arena(3) man page.
*
* Functionality defined in this header is considered as EXPERIMENTAL API.
* API standards are described in memkind(3) man page.
*/
struct memkind *get_kind_by_arena(unsigned arena_ind);
int memkind_arena_create(struct memkind *kind, struct memkind_ops *ops,
const char *name);
int memkind_arena_create_map(struct memkind *kind, extent_hooks_t *hooks);
int memkind_arena_destroy(struct memkind *kind);
void *memkind_arena_malloc(struct memkind *kind, size_t size);
void *memkind_arena_calloc(struct memkind *kind, size_t num, size_t size);
void *memkind_arena_pmem_calloc(struct memkind *kind, size_t num, size_t size);
int memkind_arena_posix_memalign(struct memkind *kind, void **memptr,
size_t alignment, size_t size);
void *memkind_arena_realloc(struct memkind *kind, void *ptr, size_t size);
int memkind_bijective_get_arena(struct memkind *kind, unsigned int *arena,
size_t size);
int memkind_thread_get_arena(struct memkind *kind, unsigned int *arena,
size_t size);
int memkind_arena_finalize(struct memkind *kind);
void memkind_arena_init(struct memkind *kind);
void memkind_arena_free(struct memkind *kind, void *ptr);
#ifdef __cplusplus
}
#endif
@@ -0,0 +1,76 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#ifndef MEMKIND_INTERNAL_API
#warning "DO NOT INCLUDE THIS FILE! IT IS INTERNAL MEMKIND API AND SOON WILL BE REMOVED FROM BIN & DEVEL PACKAGES"
#endif
#include <memkind.h>
#include <memkind/internal/memkind_private.h>
/*
* Header file for the default implementations for memkind operations.
* More details in memkind_default(3) man page.
*
* Functionality defined in this header is considered as EXPERIMENTAL API.
* API standards are described in memkind(3) man page.
*/
int memkind_default_create(struct memkind *kind, struct memkind_ops *ops,
const char *name);
int memkind_default_destroy(struct memkind *kind);
void *memkind_default_malloc(struct memkind *kind, size_t size);
void *memkind_default_calloc(struct memkind *kind, size_t num, size_t size);
int memkind_default_posix_memalign(struct memkind *kind, void **memptr,
size_t alignment, size_t size);
void *memkind_default_realloc(struct memkind *kind, void *ptr, size_t size);
void memkind_default_free(struct memkind *kind, void *ptr);
void *memkind_default_mmap(struct memkind *kind, void *addr, size_t size);
int memkind_default_mbind(struct memkind *kind, void *ptr, size_t size);
int memkind_default_get_mmap_flags(struct memkind *kind, int *flags);
int memkind_default_get_mbind_mode(struct memkind *kind, int *mode);
int memkind_default_get_mbind_nodemask(struct memkind *kind,
unsigned long *nodemask, unsigned long maxnode);
int memkind_preferred_get_mbind_mode(struct memkind *kind, int *mode);
int memkind_interleave_get_mbind_mode(struct memkind *kind, int *mode);
int memkind_nohugepage_madvise(struct memkind *kind, void *addr, size_t size);
int memkind_posix_check_alignment(struct memkind *kind, size_t alignment);
void memkind_default_init_once(void);
size_t memkind_default_malloc_usable_size(struct memkind *kind, void *ptr);
static inline bool size_out_of_bounds(size_t size)
{
return !size;
}
extern struct memkind_ops MEMKIND_DEFAULT_OPS;
#ifdef __cplusplus
}
#endif
+47
View File
@@ -0,0 +1,47 @@
/*
* Copyright (C) 2014 - 2016 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#ifndef MEMKIND_INTERNAL_API
#warning "DO NOT INCLUDE THIS FILE! IT IS INTERNAL MEMKIND API AND SOON WILL BE REMOVED FROM BIN & DEVEL PACKAGES"
#endif
#include <memkind.h>
/*
* Header file for the gigabyte TLB memkind operations.
*
* All function declarations has been moved to memkind_deprecated.h
* because of end of GBTLB support.
*
* API standards are described in memkind(3) man page.
*/
#ifdef __cplusplus
}
#endif
+70
View File
@@ -0,0 +1,70 @@
/*
* Copyright (C) 2014 - 2017 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#ifndef MEMKIND_INTERNAL_API
#warning "DO NOT INCLUDE THIS FILE! IT IS INTERNAL MEMKIND API AND SOON WILL BE REMOVED FROM BIN & DEVEL PACKAGES"
#endif
#include <memkind.h>
/*
* Header file for the high bandwidth memory memkind operations.
* More details in memkind_hbw(3) man page.
*
* Functionality defined in this header is considered as EXPERIMENTAL API.
* API standards are described in memkind(3) man page.
*/
int memkind_hbw_check_available(struct memkind *kind);
int memkind_hbw_hugetlb_check_available(struct memkind *kind);
int memkind_hbw_get_mbind_nodemask(struct memkind *kind,
unsigned long *nodemask,
unsigned long maxnode);
int memkind_hbw_all_get_mbind_nodemask(struct memkind *kind,
unsigned long *nodemask,
unsigned long maxnode);
void memkind_hbw_init_once(void);
void memkind_hbw_all_init_once(void);
void memkind_hbw_hugetlb_init_once(void);
void memkind_hbw_all_hugetlb_init_once(void);
void memkind_hbw_preferred_init_once(void);
void memkind_hbw_preferred_hugetlb_init_once(void);
void memkind_hbw_interleave_init_once(void);
extern struct memkind_ops MEMKIND_HBW_OPS;
extern struct memkind_ops MEMKIND_HBW_ALL_OPS;
extern struct memkind_ops MEMKIND_HBW_HUGETLB_OPS;
extern struct memkind_ops MEMKIND_HBW_ALL_HUGETLB_OPS;
extern struct memkind_ops MEMKIND_HBW_PREFERRED_OPS;
extern struct memkind_ops MEMKIND_HBW_PREFERRED_HUGETLB_OPS;
extern struct memkind_ops MEMKIND_HBW_INTERLEAVE_OPS;
#ifdef __cplusplus
}
#endif
+52
View File
@@ -0,0 +1,52 @@
/*
* Copyright (C) 2014 - 2017 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#ifndef MEMKIND_INTERNAL_API
#warning "DO NOT INCLUDE THIS FILE! IT IS INTERNAL MEMKIND API AND SOON WILL BE REMOVED FROM BIN & DEVEL PACKAGES"
#endif
#include <memkind.h>
/*
* Header file for the hugetlb memory memkind operations.
* More details in memkind_hugetlb(3) man page.
*
* Functionality defined in this header is considered as EXPERIMENTAL API.
* API standards are described in memkind(3) man page.
*/
int memkind_hugetlb_get_mmap_flags(struct memkind *kind, int *flags);
void memkind_hugetlb_init_once(void);
int memkind_hugetlb_check_available_2mb(struct memkind *kind);
extern struct memkind_ops MEMKIND_HUGETLB_OPS;
#ifdef __cplusplus
}
#endif
+47
View File
@@ -0,0 +1,47 @@
/*
* Copyright (C) 2015 - 2017 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#ifndef MEMKIND_INTERNAL_API
#warning "DO NOT INCLUDE THIS FILE! IT IS INTERNAL MEMKIND API AND SOON WILL BE REMOVED FROM BIN & DEVEL PACKAGES"
#endif
/*
* Header file for the interleave memory memkind operations.
*
* Functionality defined in this header is considered as EXPERIMENTAL API.
* API standards are described in memkind(3) man page.
*/
void memkind_interleave_init_once(void);
extern struct memkind_ops MEMKIND_INTERLEAVE_OPS;
#ifdef __cplusplus
}
#endif
+52
View File
@@ -0,0 +1,52 @@
/*
* Copyright (C) 2014 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#define PRINTF_FORMAT __attribute__ ((format (printf, 1, 2)))
/*
* For printing informational messages
* Requires environment variable MEMKIND_DEBUG to be set to appropriate value
*/
void log_info(const char *format, ...) PRINTF_FORMAT;
/*
* For printing messages regarding errors and failures
* Requires environment variable MEMKIND_DEBUG to be set to appropriate value
*/
void log_err(const char *format, ...) PRINTF_FORMAT;
/*
* For printing messages regarding fatal errors before calling abort()
* Works *no matter* of MEMKIND_DEBUG state
*/
void log_fatal(const char *format, ...)PRINTF_FORMAT;
#ifdef __cplusplus
}
#endif
@@ -0,0 +1,75 @@
/*
* Copyright (C) 2015 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#ifndef MEMKIND_INTERNAL_API
#warning "DO NOT INCLUDE THIS FILE! IT IS INTERNAL MEMKIND API AND SOON WILL BE REMOVED FROM BIN & DEVEL PACKAGES"
#endif
#include <memkind.h>
#include "memkind_default.h"
#include "memkind_arena.h"
#include <pthread.h>
/*
* Header file for the file-backed memory memkind operations.
* More details in memkind_pmem(3) man page.
*
* Functionality defined in this header is considered as EXPERIMENTAL API.
* API standards are described in memkind(3) man page.
*/
#define MEMKIND_PMEM_CHUNK_SIZE (1ull << 21ull) // 2MB
int memkind_pmem_create(struct memkind *kind, struct memkind_ops *ops,
const char *name);
int memkind_pmem_destroy(struct memkind *kind);
void *memkind_pmem_mmap(struct memkind *kind, void *addr, size_t size);
int memkind_pmem_get_mmap_flags(struct memkind *kind, int *flags);
struct memkind_pmem_extent {
void *addrBase;
size_t cb;
off_t offset;
};
struct memkind_pmem {
int fd;
off_t offset;
size_t max_size;
pthread_mutex_t pmem_lock;
int cextents;
int cextentsAlloc;
struct memkind_pmem_extent *rgextents;
};
extern struct memkind_ops MEMKIND_PMEM_OPS;
#ifdef __cplusplus
}
#endif
@@ -0,0 +1,127 @@
/*
* Copyright (C) 2016 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#ifndef MEMKIND_INTERNAL_API
#warning "DO NOT INCLUDE THIS FILE! IT IS INTERNAL MEMKIND API AND SOON WILL BE REMOVED FROM BIN & DEVEL PACKAGES"
#endif
#include "memkind.h"
#include <stdbool.h>
#include <pthread.h>
#ifdef __GNUC__
# define MEMKIND_LIKELY(x) __builtin_expect(!!(x), 1)
# define MEMKIND_UNLIKELY(x) __builtin_expect(!!(x), 0)
#else
# define MEMKIND_LIKELY(x) (x)
# define MEMKIND_UNLIKELY(x) (x)
#endif
#ifndef MEMKIND_EXPORT
# define MEMKIND_EXPORT __attribute__((visibility("default")))
#endif
#ifndef JE_PREFIX
#error "Can't find JE_PREFIX define. Define one or use build.sh script."
#endif
// This ladder call is required due to meanders of C's preprocessor logic.
// Without it, JE_PREFIX would be used directly (i.e. 'JE_PREFIX') and not
// substituted with defined value.
#define JE_SYMBOL2(a, b) a ## b
#define JE_SYMBOL1(a, b) JE_SYMBOL2(a, b)
#define JE_SYMBOL(b) JE_SYMBOL1(JE_PREFIX, b)
// Redefine symbols
#define jemk_malloc JE_SYMBOL(malloc)
#define jemk_mallocx JE_SYMBOL(mallocx)
#define jemk_calloc JE_SYMBOL(calloc)
#define jemk_rallocx JE_SYMBOL(rallocx)
#define jemk_realloc JE_SYMBOL(realloc)
#define jemk_mallctl JE_SYMBOL(mallctl)
#define jemk_memalign JE_SYMBOL(memalign)
#define jemk_posix_memalign JE_SYMBOL(posix_memalign)
#define jemk_free JE_SYMBOL(free)
#define jemk_dallocx JE_SYMBOL(dallocx)
#define jemk_malloc_usable_size JE_SYMBOL(malloc_usable_size)
/// \note EXPERIMENTAL API
int je_get_defrag_hint(void *ptr, int *bin_util, int *run_util);
enum memkind_const_private {
MEMKIND_NAME_LENGTH_PRIV = 64
};
struct memkind_ops {
int (* create)(struct memkind *kind, struct memkind_ops *ops, const char *name);
int (* destroy)(struct memkind *kind);
void *(* malloc)(struct memkind *kind, size_t size);
void *(* calloc)(struct memkind *kind, size_t num, size_t size);
int (* posix_memalign)(struct memkind *kind, void **memptr, size_t alignment,
size_t size);
void *(* realloc)(struct memkind *kind, void *ptr, size_t size);
void (* free)(struct memkind *kind, void *ptr);
void *(* mmap)(struct memkind *kind, void *addr, size_t size);
int (* mbind)(struct memkind *kind, void *ptr, size_t size);
int (* madvise)(struct memkind *kind, void *addr, size_t size);
int (* get_mmap_flags)(struct memkind *kind, int *flags);
int (* get_mbind_mode)(struct memkind *kind, int *mode);
int (* get_mbind_nodemask)(struct memkind *kind, unsigned long *nodemask,
unsigned long maxnode);
int (* get_arena)(struct memkind *kind, unsigned int *arena, size_t size);
int (* check_available)(struct memkind *kind);
int (* check_addr)(struct memkind *kind, void *addr);
void (* init_once)(void);
int (* finalize)(struct memkind *kind);
size_t (* malloc_usable_size)(struct memkind *kind, void *addr);
};
struct memkind {
struct memkind_ops *ops;
unsigned int partition;
char name[MEMKIND_NAME_LENGTH_PRIV];
pthread_once_t init_once;
unsigned int arena_map_len; // is power of 2
unsigned int *arena_map; // To be deleted beyond 1.2.0+
pthread_key_t arena_key;
void *priv;
unsigned int
arena_map_mask; // arena_map_len - 1 to optimize modulo operation on arena_map_len
unsigned int arena_zero; // index first jemalloc arena of this kind
};
void memkind_init(memkind_t kind, bool check_numa);
void *kind_mmap(struct memkind *kind, void *addr, size_t size);
#ifdef __cplusplus
}
#endif
+51
View File
@@ -0,0 +1,51 @@
/*
* Copyright (C) 2017 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#ifndef MEMKIND_INTERNAL_API
#warning "DO NOT INCLUDE THIS FILE! IT IS INTERNAL MEMKIND API AND SOON WILL BE REMOVED FROM BIN & DEVEL PACKAGES"
#endif
#include <memkind.h>
/*
* Header file for the regular memory memkind operations.
* More details in memkind_regular(3) man page.
*
* Functionality defined in this header is considered as EXPERIMENTAL API.
* API standards are described in memkind(3) man page.
*/
extern struct memkind_ops MEMKIND_REGULAR_OPS;
int memkind_regular_all_get_mbind_nodemask(struct memkind *kind,
unsigned long *nodemask,
unsigned long maxnode);
#ifdef __cplusplus
}
#endif
@@ -0,0 +1,34 @@
/*
Copyright (c) 2005-2018 Intel Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#include <stdint.h>
typedef void *(*rawAllocType)(intptr_t pool_id, size_t *bytes);
typedef int (*rawFreeType)(intptr_t pool_id, void *raw_ptr, size_t raw_bytes);
struct MemPoolPolicy {
rawAllocType pAlloc;
rawFreeType pFree;
size_t granularity;
int version;
unsigned fixedPool : 1,
keepAllMemory : 1,
reserved : 30;
};
+41
View File
@@ -0,0 +1,41 @@
/*
* Copyright (C) 2017 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#include <memkind.h>
#include <memkind_deprecated.h>
#ifdef __cplusplus
extern "C" {
#endif
/* ops callbacks are replaced by TBB callbacks. */
void tbb_initialize(struct memkind *kind);
/* ptr pointer must come from the valid TBB pool allocation */
void tbb_pool_free(struct memkind *kind, void *ptr);
#ifdef __cplusplus
}
#endif
+85
View File
@@ -0,0 +1,85 @@
/*
* Copyright (C) 2016 - 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/*
* !!!!!!!!!!!!!!!!!!!
* !!! WARNING !!!
* !!! PLEASE READ !!!
* !!!!!!!!!!!!!!!!!!!
*
* This header file contains all memkind deprecated symbols.
*
* Please avoid usage of this API in newly developed code, as
* eventually this code is subject for removal.
*/
#pragma once
#ifdef __cplusplus
extern "C" {
#endif
#ifndef MEMKIND_DEPRECATED
#ifdef __GNUC__
#define MEMKIND_DEPRECATED(func) func __attribute__ ((deprecated))
#elif defined(_MSC_VER)
#define MEMKIND_DEPRECATED(func) __declspec(deprecated) func
#else
#pragma message("WARNING: You need to implement MEMKIND_DEPRECATED for this compiler")
#define MEMKIND_DEPRECATED(func) func
#endif
#endif
/*
* Symbols related to GBTLB that are no longer supported
*/
extern memkind_t MEMKIND_HBW_GBTLB;
extern memkind_t MEMKIND_HBW_PREFERRED_GBTLB;
extern memkind_t MEMKIND_GBTLB;
int MEMKIND_DEPRECATED(memkind_get_kind_by_partition(int partition,
memkind_t *kind));
enum memkind_base_partition {
MEMKIND_PARTITION_DEFAULT = 0,
MEMKIND_PARTITION_HBW = 1,
MEMKIND_PARTITION_HBW_HUGETLB = 2,
MEMKIND_PARTITION_HBW_PREFERRED = 3,
MEMKIND_PARTITION_HBW_PREFERRED_HUGETLB = 4,
MEMKIND_PARTITION_HUGETLB = 5,
MEMKIND_PARTITION_HBW_GBTLB = 6,
MEMKIND_PARTITION_HBW_PREFERRED_GBTLB = 7,
MEMKIND_PARTITION_GBTLB = 8,
MEMKIND_PARTITION_HBW_INTERLEAVE = 9,
MEMKIND_PARTITION_INTERLEAVE = 10,
MEMKIND_PARTITION_REGULAR = 11,
MEMKIND_PARTITION_HBW_ALL = 12,
MEMKIND_PARTITION_HBW_ALL_HUGETLB = 13,
MEMKIND_NUM_BASE_KIND
};
#ifdef __cplusplus
}
#endif
+201
View File
@@ -0,0 +1,201 @@
/*
* Copyright (C) 2018 Intel Corporation.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* 1. Redistributions of source code must retain the above copyright notice(s),
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice(s),
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
* EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#include <memory>
#include <string>
#include <exception>
#include <type_traits>
#include <atomic>
#include <cstddef>
#include "memkind.h"
/*
* Header file for the C++ allocator compatible with the C++ standard library allocator concepts.
* More details in pmemallocator(3) man page.
* Note: memory heap management is based on memkind_malloc, refer to the memkind(3) man page for more
* information.
*
* Functionality defined in this header is considered as EXPERIMENTAL API.
* API standards are described in memkind(3) man page.
*/
namespace pmem
{
namespace internal
{
class kind_wrapper_t
{
public:
kind_wrapper_t(const char *dir, std::size_t max_size)
{
int err_c = memkind_create_pmem(dir, max_size, &kind);
if (err_c) {
throw std::runtime_error(
std::string("An error occured while creating pmem kind; error code: ") +
std::to_string(err_c));
}
}
kind_wrapper_t(const kind_wrapper_t &) = delete;
void operator=(const kind_wrapper_t &) = delete;
~kind_wrapper_t()
{
memkind_destroy_kind(kind);
}
memkind_t get() const
{
return kind;
}
private:
memkind_t kind;
};
}
template<typename T>
class allocator
{
using kind_wrapper_t = internal::kind_wrapper_t;
std::shared_ptr<kind_wrapper_t> kind_wrapper_ptr;
public:
using value_type = T;
using pointer = value_type*;
using const_pointer = const value_type*;
using reference = value_type&;
using const_reference = const value_type&;
using size_type = size_t;
using difference_type = ptrdiff_t;
template<class U>
struct rebind {
using other = allocator<U>;
};
template<typename U>
friend class allocator;
#ifndef _GLIBCXX_USE_CXX11_ABI
/* This is a workaround for compilers (e.g GCC 4.8) that uses C++11 standard,
* but use old - non C++11 ABI */
template<typename V = void>
explicit allocator()
{
static_assert(std::is_same<V, void>::value,
"pmem::allocator cannot be compiled without CXX11 ABI");
}
#endif
explicit allocator(const char *dir, size_t max_size) :
kind_wrapper_ptr(std::make_shared<kind_wrapper_t>(dir, max_size))
{
}
explicit allocator(const std::string &dir, size_t max_size) :
allocator(dir.c_str(), max_size)
{
}
allocator(const allocator &other) = default;
template <typename U>
allocator(const allocator<U> &other) noexcept : kind_wrapper_ptr(
other.kind_wrapper_ptr)
{
}
allocator(allocator &&other) = default;
template <typename U>
allocator(allocator<U> &&other) noexcept :
kind_wrapper_ptr(std::move(other.kind_wrapper_ptr))
{
}
allocator<T> &operator = (const allocator &other) = default;
template <typename U>
allocator<T> &operator = (const allocator<U> &other) noexcept
{
kind_wrapper_ptr = other.kind_wrapper_ptr;
return *this;
}
allocator<T> &operator = (allocator &&other) = default;
template <typename U>
allocator<T> &operator = (allocator<U> &&other) noexcept
{
kind_wrapper_ptr = std::move(other.kind_wrapper_ptr);
return *this;
}
pointer allocate(size_type n) const
{
pointer result = static_cast<pointer>(memkind_malloc(kind_wrapper_ptr->get(),
n*sizeof(T)));
if (!result) {
throw std::bad_alloc();
}
return result;
}
void deallocate(pointer p, size_type n) const
{
memkind_free(kind_wrapper_ptr->get(), static_cast<void *>(p));
}
template <class U, class... Args>
void construct(U *p, Args &&... args) const
{
::new((void *)p) U(std::forward<Args>(args)...);
}
void destroy(pointer p) const
{
p->~value_type();
}
template <typename U, typename V>
friend bool operator ==(const allocator<U> &lhs, const allocator<V> &rhs);
template <typename U, typename V>
friend bool operator !=(const allocator<U> &lhs, const allocator<V> &rhs);
};
template <typename U, typename V>
bool operator ==(const allocator<U> &lhs, const allocator<V> &rhs)
{
return lhs.kind_wrapper_ptr->get() == rhs.kind_wrapper_ptr->get();
}
template <typename U, typename V>
bool operator !=(const allocator<U> &lhs, const allocator<V> &rhs)
{
return !(lhs == rhs);
}
}
+31
View File
@@ -0,0 +1,31 @@
#!/bin/bash
#
# Copyright (C) 2018 Intel Corporation.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice(s),
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice(s),
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
set -ex
curl -SL https://sourceforge.net/projects/astyle/files/astyle/astyle%203.1/astyle_3.1_linux.tar.gz -o /tmp/astyle.tar.gz
tar -xzvf /tmp/astyle.tar.gz -C /tmp/
echo "Install astyle"
cd /tmp/astyle/build/gcc
make
sudo make install
+42
View File
@@ -0,0 +1,42 @@
version: '{build}'
environment:
matrix:
- MSYSTEM: MINGW64
CPU: x86_64
MSVC: amd64
- MSYSTEM: MINGW32
CPU: i686
MSVC: x86
- MSYSTEM: MINGW64
CPU: x86_64
- MSYSTEM: MINGW32
CPU: i686
- MSYSTEM: MINGW64
CPU: x86_64
MSVC: amd64
CONFIG_FLAGS: --enable-debug
- MSYSTEM: MINGW32
CPU: i686
MSVC: x86
CONFIG_FLAGS: --enable-debug
- MSYSTEM: MINGW64
CPU: x86_64
CONFIG_FLAGS: --enable-debug
- MSYSTEM: MINGW32
CPU: i686
CONFIG_FLAGS: --enable-debug
install:
- set PATH=c:\msys64\%MSYSTEM%\bin;c:\msys64\usr\bin;%PATH%
- if defined MSVC call "c:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" %MSVC%
- if defined MSVC pacman --noconfirm -Rsc mingw-w64-%CPU%-gcc gcc
- pacman --noconfirm -Suy mingw-w64-%CPU%-make
build_script:
- bash -c "autoconf"
- bash -c "./configure $CONFIG_FLAGS"
- mingw32-make
- file lib/jemalloc.dll
- mingw32-make tests
- mingw32-make -k check
+3
View File
@@ -0,0 +1,3 @@
begin-language: "Autoconf-without-aclocal-m4"
args: --no-cache
end-language: "Autoconf-without-aclocal-m4"
+1
View File
@@ -0,0 +1 @@
* text=auto eol=lf
+91
View File
@@ -0,0 +1,91 @@
/bin/jemalloc-config
/bin/jemalloc.sh
/bin/jeprof
/config.stamp
/config.log
/config.status
/configure
/doc/html.xsl
/doc/manpages.xsl
/doc/jemalloc.xml
/doc/jemalloc.html
/doc/jemalloc.3
/jemalloc.pc
/lib/
/Makefile
/include/jemalloc/internal/jemalloc_preamble.h
/include/jemalloc/internal/jemalloc_internal_defs.h
/include/jemalloc/internal/private_namespace.gen.h
/include/jemalloc/internal/private_namespace.h
/include/jemalloc/internal/private_namespace_jet.gen.h
/include/jemalloc/internal/private_namespace_jet.h
/include/jemalloc/internal/private_symbols.awk
/include/jemalloc/internal/private_symbols_jet.awk
/include/jemalloc/internal/public_namespace.h
/include/jemalloc/internal/public_symbols.txt
/include/jemalloc/internal/public_unnamespace.h
/include/jemalloc/internal/size_classes.h
/include/jemalloc/jemalloc.h
/include/jemalloc/jemalloc_defs.h
/include/jemalloc/jemalloc_macros.h
/include/jemalloc/jemalloc_mangle.h
/include/jemalloc/jemalloc_mangle_jet.h
/include/jemalloc/jemalloc_protos.h
/include/jemalloc/jemalloc_protos_jet.h
/include/jemalloc/jemalloc_rename.h
/include/jemalloc/jemalloc_typedefs.h
/src/*.[od]
/src/*.sym
/run_tests.out/
/test/test.sh
test/include/test/jemalloc_test.h
test/include/test/jemalloc_test_defs.h
/test/integration/[A-Za-z]*
!/test/integration/[A-Za-z]*.*
/test/integration/*.[od]
/test/integration/*.out
/test/integration/cpp/[A-Za-z]*
!/test/integration/cpp/[A-Za-z]*.*
/test/integration/cpp/*.[od]
/test/integration/cpp/*.out
/test/src/*.[od]
/test/stress/[A-Za-z]*
!/test/stress/[A-Za-z]*.*
/test/stress/*.[od]
/test/stress/*.out
/test/unit/[A-Za-z]*
!/test/unit/[A-Za-z]*.*
/test/unit/*.[od]
/test/unit/*.out
/VERSION
*.pdb
*.sdf
*.opendb
*.opensdf
*.cachefile
*.suo
*.user
*.sln.docstates
*.tmp
/msvc/Win32/
/msvc/x64/
/msvc/projects/*/*/Debug*/
/msvc/projects/*/*/Release*/
/msvc/projects/*/*/Win32/
/msvc/projects/*/*/x64/
+155
View File
@@ -0,0 +1,155 @@
language: generic
matrix:
include:
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: osx
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=clang CXX=clang++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--enable-debug" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--enable-prof" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="-m32" CONFIGURE_FLAGS="--with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
addons:
apt:
packages:
- gcc-multilib
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --enable-prof" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-debug --with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof --disable-stats" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof --with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof --with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof --with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--enable-prof --with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats --with-malloc-conf=tcache:false" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats --with-malloc-conf=dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats --with-malloc-conf=percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--disable-stats --with-malloc-conf=background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false,dss:primary" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false,percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=tcache:false,background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=dss:primary,percpu_arena:percpu" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=dss:primary,background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
- os: linux
env: CC=gcc CXX=g++ COMPILER_FLAGS="" CONFIGURE_FLAGS="--with-malloc-conf=percpu_arena:percpu,background_thread:true" EXTRA_CFLAGS="-Werror -Wno-array-bounds"
before_script:
- autoconf
- ./configure ${COMPILER_FLAGS:+ CC="$CC $COMPILER_FLAGS" CXX="$CXX $COMPILER_FLAGS" } $CONFIGURE_FLAGS
- make -j3
- make -j3 tests
script:
- make check
+27
View File
@@ -0,0 +1,27 @@
Unless otherwise specified, files in the jemalloc source distribution are
subject to the following license:
--------------------------------------------------------------------------------
Copyright (C) 2002-2017 Jason Evans <jasone@canonware.com>.
All rights reserved.
Copyright (C) 2007-2012 Mozilla Foundation. All rights reserved.
Copyright (C) 2009-2017 Facebook, Inc. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice(s),
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice(s),
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
--------------------------------------------------------------------------------
File diff suppressed because it is too large. Load diff
+411
View File
@@ -0,0 +1,411 @@
Building and installing a packaged release of jemalloc can be as simple as
typing the following while in the root directory of the source tree:
./configure
make
make install
If building from unpackaged developer sources, the simplest command sequence
that might work is:
./autogen.sh
make dist
make
make install
Note that documentation is not built by the default target because doing so
would create a dependency on xsltproc in packaged releases, hence the
requirement to either run 'make dist' or avoid installing docs via the various
install_* targets documented below.
## Advanced configuration
The 'configure' script supports numerous options that allow control of which
functionality is enabled, where jemalloc is installed, etc. Optionally, pass
any of the following arguments (not a definitive list) to 'configure':
* `--help`
Print a definitive list of options.
* `--prefix=<install-root-dir>`
Set the base directory in which to install. For example:
./configure --prefix=/usr/local
will cause files to be installed into /usr/local/include, /usr/local/lib,
and /usr/local/man.
* `--with-version=(<major>.<minor>.<bugfix>-<nrev>-g<gid>|VERSION)`
The VERSION file is mandatory for successful configuration, and the
following steps are taken to assure its presence:
1) If --with-version=<major>.<minor>.<bugfix>-<nrev>-g<gid> is specified,
generate VERSION using the specified value.
2) If --with-version is not specified in either form and the source
directory is inside a git repository, try to generate VERSION via 'git
describe' invocations that pattern-match release tags.
3) If VERSION is missing, generate it with a bogus version:
0.0.0-0-g0000000000000000000000000000000000000000
Note that --with-version=VERSION bypasses (1) and (2), which simplifies
VERSION configuration when embedding a jemalloc release into another
project's git repository.
* `--with-rpath=<colon-separated-rpath>`
Embed one or more library paths, so that libjemalloc can find the libraries
it is linked to. This works only on ELF-based systems.
* `--with-mangling=<map>`
Mangle public symbols specified in <map> which is a comma-separated list of
name:mangled pairs.
For example, to use ld's --wrap option as an alternative method for
overriding libc's malloc implementation, specify something like:
--with-mangling=malloc:__wrap_malloc,free:__wrap_free[...]
Note that mangling happens prior to application of the prefix specified by
--with-jemalloc-prefix, and mangled symbols are then ignored when applying
the prefix.
* `--with-jemalloc-prefix=<prefix>`
Prefix all public APIs with <prefix>. For example, if <prefix> is
"prefix_", API changes like the following occur:
malloc() --> prefix_malloc()
malloc_conf --> prefix_malloc_conf
/etc/malloc.conf --> /etc/prefix_malloc.conf
MALLOC_CONF --> PREFIX_MALLOC_CONF
This makes it possible to use jemalloc at the same time as the system
allocator, or even to use multiple copies of jemalloc simultaneously.
By default, the prefix is "", except on OS X, where it is "je_". On OS X,
jemalloc overlays the default malloc zone, but makes no attempt to actually
replace the "malloc", "calloc", etc. symbols.
* `--without-export`
Don't export public APIs. This can be useful when building jemalloc as a
static library, or to avoid exporting public APIs when using the zone
allocator on OSX.
* `--with-private-namespace=<prefix>`
Prefix all library-private APIs with <prefix>je_. For shared libraries,
symbol visibility mechanisms prevent these symbols from being exported, but
for static libraries, naming collisions are a real possibility. By
default, <prefix> is empty, which results in a symbol prefix of je_ .
* `--with-install-suffix=<suffix>`
Append <suffix> to the base name of all installed files, such that multiple
versions of jemalloc can coexist in the same installation directory. For
example, libjemalloc.so.0 becomes libjemalloc<suffix>.so.0.
* `--with-malloc-conf=<malloc_conf>`
Embed `<malloc_conf>` as a run-time options string that is processed prior to
the malloc_conf global variable, the /etc/malloc.conf symlink, and the
MALLOC_CONF environment variable. For example, to change the default decay
time to 30 seconds:
--with-malloc-conf=decay_ms:30000
* `--enable-debug`
Enable assertions and validation code. This incurs a substantial
performance hit, but is very useful during application development.
* `--disable-stats`
Disable statistics gathering functionality. See the "opt.stats_print"
option documentation for usage details.
* `--enable-prof`
Enable heap profiling and leak detection functionality. See the "opt.prof"
option documentation for usage details. When enabled, there are several
approaches to backtracing, and the configure script chooses the first one
in the following list that appears to function correctly:
+ libunwind (requires --enable-prof-libunwind)
+ libgcc (unless --disable-prof-libgcc)
+ gcc intrinsics (unless --disable-prof-gcc)
* `--enable-prof-libunwind`
Use the libunwind library (http://www.nongnu.org/libunwind/) for stack
backtracing.
* `--disable-prof-libgcc`
Disable the use of libgcc's backtracing functionality.
* `--disable-prof-gcc`
Disable the use of gcc intrinsics for backtracing.
* `--with-static-libunwind=<libunwind.a>`
Statically link against the specified libunwind.a rather than dynamically
linking with -lunwind.
* `--disable-thp`
Disable transparent huge page (THP) integration. This option can be useful
when cross compiling.
* `--disable-fill`
Disable support for junk/zero filling of memory. See the "opt.junk" and
"opt.zero" option documentation for usage details.
* `--disable-zone-allocator`
Disable zone allocator for Darwin. This means jemalloc won't be hooked as
the default allocator on OSX/iOS.
* `--enable-utrace`
Enable utrace(2)-based allocation tracing. This feature is not broadly
portable (FreeBSD has it, but Linux and OS X do not).
* `--enable-xmalloc`
Enable support for optional immediate termination due to out-of-memory
errors, as is commonly implemented by "xmalloc" wrapper function for malloc.
See the "opt.xmalloc" option documentation for usage details.
* `--enable-lazy-lock`
Enable code that wraps pthread_create() to detect when an application
switches from single-threaded to multi-threaded mode, so that it can avoid
mutex locking/unlocking operations while in single-threaded mode. In
practice, this feature usually has little impact on performance unless
thread-specific caching is disabled.
* `--disable-cache-oblivious`
Disable cache-oblivious large allocation alignment for large allocation
requests with no alignment constraints. If this feature is disabled, all
large allocations are page-aligned as an implementation artifact, which can
severely harm CPU cache utilization. However, the cache-oblivious layout
comes at the cost of one extra page per large allocation, which in the
most extreme case increases physical memory usage for the 16 KiB size class
to 20 KiB.
* `--disable-syscall`
Disable use of syscall(2) rather than {open,read,write,close}(2). This is
intended as a workaround for systems that place security limitations on
syscall(2).
* `--disable-cxx`
Disable C++ integration. This will cause new and delete operator
implementations to be omitted.
* `--with-xslroot=<path>`
Specify where to find DocBook XSL stylesheets when building the
documentation.
* `--with-lg-page=<lg-page>`
Specify the base 2 log of the allocator page size, which must in turn be at
least as large as the system page size. By default the configure script
determines the host's page size and sets the allocator page size equal to
the system page size, so this option need not be specified unless the
system page size may change between configuration and execution, e.g. when
cross compiling.
* `--with-lg-page-sizes=<lg-page-sizes>`
Specify the comma-separated base 2 logs of the page sizes to support. This
option may be useful when cross compiling in combination with
`--with-lg-page`, but its primary use case is for integration with FreeBSD's
libc, wherein jemalloc is embedded.
* `--with-lg-hugepage=<lg-hugepage>`
Specify the base 2 log of the system huge page size. This option is useful
when cross compiling, or when overriding the default for systems that do
not explicitly support huge pages.
* `--with-lg-quantum=<lg-quantum>`
Specify the base 2 log of the minimum allocation alignment. jemalloc needs
to know the minimum alignment that meets the following C standard
requirement (quoted from the April 12, 2011 draft of the C11 standard):
> The pointer returned if the allocation succeeds is suitably aligned so
that it may be assigned to a pointer to any type of object with a
fundamental alignment requirement and then used to access such an object
or an array of such objects in the space allocated [...]
This setting is architecture-specific, and although jemalloc includes known
safe values for the most commonly used modern architectures, there is a
wrinkle related to GNU libc (glibc) that may impact your choice of
<lg-quantum>. On most modern architectures, this mandates 16-byte
alignment (<lg-quantum>=4), but the glibc developers chose not to meet this
requirement for performance reasons. An old discussion can be found at
<https://sourceware.org/bugzilla/show_bug.cgi?id=206> . Unlike glibc,
jemalloc does follow the C standard by default (caveat: jemalloc
technically cheats for size classes smaller than the quantum), but the fact
that Linux systems already work around this allocator noncompliance means
that it is generally safe in practice to let jemalloc's minimum alignment
follow glibc's lead. If you specify `--with-lg-quantum=3` during
configuration, jemalloc will provide additional size classes that are not
16-byte-aligned (24, 40, and 56).
The following environment variables (not a definitive list) impact configure's
behavior:
* `CFLAGS="?"`
* `CXXFLAGS="?"`
Pass these flags to the C/C++ compiler. Any flags set by the configure
script are prepended, which means explicitly set flags generally take
precedence. Take care when specifying flags such as -Werror, because
configure tests may be affected in undesirable ways.
* `EXTRA_CFLAGS="?"`
* `EXTRA_CXXFLAGS="?"`
Append these flags to CFLAGS/CXXFLAGS, without passing them to the
compiler(s) during configuration. This makes it possible to add flags such
as -Werror, while allowing the configure script to determine what other
flags are appropriate for the specified configuration.
* `CPPFLAGS="?"`
Pass these flags to the C preprocessor. Note that CFLAGS is not passed to
'cpp' when 'configure' is looking for include files, so you must use
CPPFLAGS instead if you need to help 'configure' find header files.
* `LD_LIBRARY_PATH="?"`
'ld' uses this colon-separated list to find libraries.
* `LDFLAGS="?"`
Pass these flags when linking.
* `PATH="?"`
'configure' uses this to find programs.
In some cases it may be necessary to work around configuration results that do
not match reality. For example, Linux 4.5 added support for the MADV_FREE flag
to madvise(2), which can cause problems if building on a host with MADV_FREE
support and deploying to a target without. To work around this, use a cache
file to override the relevant configuration variable defined in configure.ac,
e.g.:
echo "je_cv_madv_free=no" > config.cache && ./configure -C
## Advanced compilation
To build only parts of jemalloc, use the following targets:
build_lib_shared
build_lib_static
build_lib
build_doc_html
build_doc_man
build_doc
To install only parts of jemalloc, use the following targets:
install_bin
install_include
install_lib_shared
install_lib_static
install_lib
install_doc_html
install_doc_man
install_doc
To clean up build results to varying degrees, use the following make targets:
clean
distclean
relclean
## Advanced installation
Optionally, define make variables when invoking make, including (not
exclusively):
* `INCLUDEDIR="?"`
Use this as the installation prefix for header files.
* `LIBDIR="?"`
Use this as the installation prefix for libraries.
* `MANDIR="?"`
Use this as the installation prefix for man pages.
* `DESTDIR="?"`
Prepend DESTDIR to INCLUDEDIR, LIBDIR, DATADIR, and MANDIR. This is useful
when installing to a different path than was specified via --prefix.
* `CC="?"`
Use this to invoke the C compiler.
* `CFLAGS="?"`
Pass these flags to the compiler.
* `CPPFLAGS="?"`
Pass these flags to the C preprocessor.
* `LDFLAGS="?"`
Pass these flags when linking.
* `PATH="?"`
Use this to search for programs used during configuration and building.
## Development
If you intend to make non-trivial changes to jemalloc, use the 'autogen.sh'
script rather than 'configure'. This re-generates 'configure', enables
configuration dependency rules, and enables re-generation of automatically
generated source files.
The build system supports using an object directory separate from the source
tree. For example, you can create an 'obj' directory, and from within that
directory, issue configuration and build commands:
autoconf
mkdir obj
cd obj
../configure --enable-autogen
make
## Documentation
The manual page is generated in both html and roff formats. Any web browser
can be used to view the html manual. The roff manual page can be formatted
prior to installation via the following command:
nroff -man -t doc/jemalloc.3
+563
View File
@@ -0,0 +1,563 @@
# Clear out all vpaths, then set just one (default vpath) for the main build
# directory.
vpath
vpath % .
# Clear the default suffixes, so that built-in rules are not used.
.SUFFIXES :
SHELL := /bin/sh
CC := @CC@
CXX := @CXX@
# Configuration parameters.
DESTDIR =
BINDIR := $(DESTDIR)@BINDIR@
INCLUDEDIR := $(DESTDIR)@INCLUDEDIR@
LIBDIR := $(DESTDIR)@LIBDIR@
DATADIR := $(DESTDIR)@DATADIR@
MANDIR := $(DESTDIR)@MANDIR@
srcroot := @srcroot@
objroot := @objroot@
abs_srcroot := @abs_srcroot@
abs_objroot := @abs_objroot@
# Build parameters.
CPPFLAGS := @CPPFLAGS@ -I$(srcroot)include -I$(objroot)include
CONFIGURE_CFLAGS := @CONFIGURE_CFLAGS@
SPECIFIED_CFLAGS := @SPECIFIED_CFLAGS@
EXTRA_CFLAGS := @EXTRA_CFLAGS@
CFLAGS := $(strip $(CONFIGURE_CFLAGS) $(SPECIFIED_CFLAGS) $(EXTRA_CFLAGS))
CONFIGURE_CXXFLAGS := @CONFIGURE_CXXFLAGS@
SPECIFIED_CXXFLAGS := @SPECIFIED_CXXFLAGS@
EXTRA_CXXFLAGS := @EXTRA_CXXFLAGS@
CXXFLAGS := $(strip $(CONFIGURE_CXXFLAGS) $(SPECIFIED_CXXFLAGS) $(EXTRA_CXXFLAGS))
LDFLAGS := @LDFLAGS@
EXTRA_LDFLAGS := @EXTRA_LDFLAGS@
LIBS := @LIBS@
RPATH_EXTRA := @RPATH_EXTRA@
SO := @so@
IMPORTLIB := @importlib@
O := @o@
A := @a@
EXE := @exe@
LIBPREFIX := @libprefix@
REV := @rev@
install_suffix := @install_suffix@
ABI := @abi@
XSLTPROC := @XSLTPROC@
AUTOCONF := @AUTOCONF@
_RPATH = @RPATH@
RPATH = $(if $(1),$(call _RPATH,$(1)))
cfghdrs_in := $(addprefix $(srcroot),@cfghdrs_in@)
cfghdrs_out := @cfghdrs_out@
cfgoutputs_in := $(addprefix $(srcroot),@cfgoutputs_in@)
cfgoutputs_out := @cfgoutputs_out@
enable_autogen := @enable_autogen@
enable_prof := @enable_prof@
enable_zone_allocator := @enable_zone_allocator@
MALLOC_CONF := @JEMALLOC_CPREFIX@MALLOC_CONF
link_whole_archive := @link_whole_archive@
DSO_LDFLAGS = @DSO_LDFLAGS@
SOREV = @SOREV@
PIC_CFLAGS = @PIC_CFLAGS@
CTARGET = @CTARGET@
LDTARGET = @LDTARGET@
TEST_LD_MODE = @TEST_LD_MODE@
MKLIB = @MKLIB@
AR = @AR@
ARFLAGS = @ARFLAGS@
DUMP_SYMS = @DUMP_SYMS@
AWK := @AWK@
CC_MM = @CC_MM@
LM := @LM@
INSTALL = @INSTALL@
ifeq (macho, $(ABI))
TEST_LIBRARY_PATH := DYLD_FALLBACK_LIBRARY_PATH="$(objroot)lib"
else
ifeq (pecoff, $(ABI))
TEST_LIBRARY_PATH := PATH="$(PATH):$(objroot)lib"
else
TEST_LIBRARY_PATH :=
endif
endif
LIBJEMALLOC := $(LIBPREFIX)jemalloc$(install_suffix)
# Lists of files.
BINS := $(objroot)bin/jemalloc-config $(objroot)bin/jemalloc.sh $(objroot)bin/jeprof
C_HDRS := $(objroot)include/jemalloc/jemalloc$(install_suffix).h
C_SRCS := $(srcroot)src/jemalloc.c \
$(srcroot)src/arena.c \
$(srcroot)src/background_thread.c \
$(srcroot)src/base.c \
$(srcroot)src/bitmap.c \
$(srcroot)src/ckh.c \
$(srcroot)src/ctl.c \
$(srcroot)src/extent.c \
$(srcroot)src/extent_dss.c \
$(srcroot)src/extent_mmap.c \
$(srcroot)src/hash.c \
$(srcroot)src/hooks.c \
$(srcroot)src/large.c \
$(srcroot)src/malloc_io.c \
$(srcroot)src/mutex.c \
$(srcroot)src/mutex_pool.c \
$(srcroot)src/nstime.c \
$(srcroot)src/pages.c \
$(srcroot)src/prng.c \
$(srcroot)src/prof.c \
$(srcroot)src/rtree.c \
$(srcroot)src/stats.c \
$(srcroot)src/spin.c \
$(srcroot)src/sz.c \
$(srcroot)src/tcache.c \
$(srcroot)src/ticker.c \
$(srcroot)src/tsd.c \
$(srcroot)src/witness.c
ifeq ($(enable_zone_allocator), 1)
C_SRCS += $(srcroot)src/zone.c
endif
ifeq ($(IMPORTLIB),$(SO))
STATIC_LIBS := $(objroot)lib/$(LIBJEMALLOC).$(A)
endif
ifdef PIC_CFLAGS
STATIC_LIBS += $(objroot)lib/$(LIBJEMALLOC)_pic.$(A)
else
STATIC_LIBS += $(objroot)lib/$(LIBJEMALLOC)_s.$(A)
endif
DSOS := $(objroot)lib/$(LIBJEMALLOC).$(SOREV)
ifneq ($(SOREV),$(SO))
DSOS += $(objroot)lib/$(LIBJEMALLOC).$(SO)
endif
ifeq (1, $(link_whole_archive))
LJEMALLOC := -Wl,--whole-archive -L$(objroot)lib -l$(LIBJEMALLOC) -Wl,--no-whole-archive
else
LJEMALLOC := $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB)
endif
PC := $(objroot)jemalloc.pc
MAN3 := $(objroot)doc/jemalloc$(install_suffix).3
DOCS_XML := $(objroot)doc/jemalloc$(install_suffix).xml
DOCS_HTML := $(DOCS_XML:$(objroot)%.xml=$(objroot)%.html)
DOCS_MAN3 := $(DOCS_XML:$(objroot)%.xml=$(objroot)%.3)
DOCS := $(DOCS_HTML) $(DOCS_MAN3)
C_TESTLIB_SRCS := $(srcroot)test/src/btalloc.c $(srcroot)test/src/btalloc_0.c \
$(srcroot)test/src/btalloc_1.c $(srcroot)test/src/math.c \
$(srcroot)test/src/mtx.c $(srcroot)test/src/mq.c \
$(srcroot)test/src/SFMT.c $(srcroot)test/src/test.c \
$(srcroot)test/src/thd.c $(srcroot)test/src/timer.c
ifeq (1, $(link_whole_archive))
C_UTIL_INTEGRATION_SRCS :=
C_UTIL_CPP_SRCS :=
else
C_UTIL_INTEGRATION_SRCS := $(srcroot)src/nstime.c $(srcroot)src/malloc_io.c
C_UTIL_CPP_SRCS := $(srcroot)src/nstime.c $(srcroot)src/malloc_io.c
endif
TESTS_UNIT := \
$(srcroot)test/unit/a0.c \
$(srcroot)test/unit/arena_reset.c \
$(srcroot)test/unit/atomic.c \
$(srcroot)test/unit/background_thread.c \
$(srcroot)test/unit/base.c \
$(srcroot)test/unit/bitmap.c \
$(srcroot)test/unit/ckh.c \
$(srcroot)test/unit/decay.c \
$(srcroot)test/unit/extent_quantize.c \
$(srcroot)test/unit/fork.c \
$(srcroot)test/unit/hash.c \
$(srcroot)test/unit/hooks.c \
$(srcroot)test/unit/junk.c \
$(srcroot)test/unit/junk_alloc.c \
$(srcroot)test/unit/junk_free.c \
$(srcroot)test/unit/mallctl.c \
$(srcroot)test/unit/malloc_io.c \
$(srcroot)test/unit/math.c \
$(srcroot)test/unit/mq.c \
$(srcroot)test/unit/mtx.c \
$(srcroot)test/unit/pack.c \
$(srcroot)test/unit/pages.c \
$(srcroot)test/unit/ph.c \
$(srcroot)test/unit/prng.c \
$(srcroot)test/unit/prof_accum.c \
$(srcroot)test/unit/prof_active.c \
$(srcroot)test/unit/prof_gdump.c \
$(srcroot)test/unit/prof_idump.c \
$(srcroot)test/unit/prof_reset.c \
$(srcroot)test/unit/prof_tctx.c \
$(srcroot)test/unit/prof_thread_name.c \
$(srcroot)test/unit/ql.c \
$(srcroot)test/unit/qr.c \
$(srcroot)test/unit/rb.c \
$(srcroot)test/unit/retained.c \
$(srcroot)test/unit/rtree.c \
$(srcroot)test/unit/SFMT.c \
$(srcroot)test/unit/size_classes.c \
$(srcroot)test/unit/slab.c \
$(srcroot)test/unit/smoothstep.c \
$(srcroot)test/unit/spin.c \
$(srcroot)test/unit/stats.c \
$(srcroot)test/unit/stats_print.c \
$(srcroot)test/unit/ticker.c \
$(srcroot)test/unit/nstime.c \
$(srcroot)test/unit/tsd.c \
$(srcroot)test/unit/witness.c \
$(srcroot)test/unit/zero.c
ifeq (@enable_prof@, 1)
TESTS_UNIT += \
$(srcroot)test/unit/arena_reset_prof.c
endif
TESTS_INTEGRATION := $(srcroot)test/integration/aligned_alloc.c \
$(srcroot)test/integration/allocated.c \
$(srcroot)test/integration/extent.c \
$(srcroot)test/integration/mallocx.c \
$(srcroot)test/integration/MALLOCX_ARENA.c \
$(srcroot)test/integration/overflow.c \
$(srcroot)test/integration/posix_memalign.c \
$(srcroot)test/integration/rallocx.c \
$(srcroot)test/integration/sdallocx.c \
$(srcroot)test/integration/thread_arena.c \
$(srcroot)test/integration/thread_tcache_enabled.c \
$(srcroot)test/integration/xallocx.c
ifeq (@enable_cxx@, 1)
CPP_SRCS := $(srcroot)src/jemalloc_cpp.cpp
TESTS_INTEGRATION_CPP := $(srcroot)test/integration/cpp/basic.cpp
else
CPP_SRCS :=
TESTS_INTEGRATION_CPP :=
endif
TESTS_STRESS := $(srcroot)test/stress/microbench.c
TESTS := $(TESTS_UNIT) $(TESTS_INTEGRATION) $(TESTS_INTEGRATION_CPP) $(TESTS_STRESS)
PRIVATE_NAMESPACE_HDRS := $(objroot)include/jemalloc/internal/private_namespace.h $(objroot)include/jemalloc/internal/private_namespace_jet.h
PRIVATE_NAMESPACE_GEN_HDRS := $(PRIVATE_NAMESPACE_HDRS:%.h=%.gen.h)
C_SYM_OBJS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.sym.$(O))
C_SYMS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.sym)
C_OBJS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.$(O))
CPP_OBJS := $(CPP_SRCS:$(srcroot)%.cpp=$(objroot)%.$(O))
C_PIC_OBJS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.pic.$(O))
CPP_PIC_OBJS := $(CPP_SRCS:$(srcroot)%.cpp=$(objroot)%.pic.$(O))
C_JET_SYM_OBJS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.jet.sym.$(O))
C_JET_SYMS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.jet.sym)
C_JET_OBJS := $(C_SRCS:$(srcroot)%.c=$(objroot)%.jet.$(O))
C_TESTLIB_UNIT_OBJS := $(C_TESTLIB_SRCS:$(srcroot)%.c=$(objroot)%.unit.$(O))
C_TESTLIB_INTEGRATION_OBJS := $(C_TESTLIB_SRCS:$(srcroot)%.c=$(objroot)%.integration.$(O))
C_UTIL_INTEGRATION_OBJS := $(C_UTIL_INTEGRATION_SRCS:$(srcroot)%.c=$(objroot)%.integration.$(O))
C_TESTLIB_STRESS_OBJS := $(C_TESTLIB_SRCS:$(srcroot)%.c=$(objroot)%.stress.$(O))
C_TESTLIB_OBJS := $(C_TESTLIB_UNIT_OBJS) $(C_TESTLIB_INTEGRATION_OBJS) $(C_UTIL_INTEGRATION_OBJS) $(C_TESTLIB_STRESS_OBJS)
TESTS_UNIT_OBJS := $(TESTS_UNIT:$(srcroot)%.c=$(objroot)%.$(O))
TESTS_INTEGRATION_OBJS := $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%.$(O))
TESTS_INTEGRATION_CPP_OBJS := $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%.$(O))
TESTS_STRESS_OBJS := $(TESTS_STRESS:$(srcroot)%.c=$(objroot)%.$(O))
TESTS_OBJS := $(TESTS_UNIT_OBJS) $(TESTS_INTEGRATION_OBJS) $(TESTS_STRESS_OBJS)
TESTS_CPP_OBJS := $(TESTS_INTEGRATION_CPP_OBJS)
.PHONY: all dist build_doc_html build_doc_man build_doc
.PHONY: install_bin install_include install_lib
.PHONY: install_doc_html install_doc_man install_doc install
.PHONY: tests check clean distclean relclean
.SECONDARY : $(PRIVATE_NAMESPACE_GEN_HDRS) $(TESTS_OBJS) $(TESTS_CPP_OBJS)
# Default target.
all: build_lib
dist: build_doc
$(objroot)doc/%.html : $(objroot)doc/%.xml $(srcroot)doc/stylesheet.xsl $(objroot)doc/html.xsl
$(XSLTPROC) -o $@ $(objroot)doc/html.xsl $<
$(objroot)doc/%.3 : $(objroot)doc/%.xml $(srcroot)doc/stylesheet.xsl $(objroot)doc/manpages.xsl
$(XSLTPROC) -o $@ $(objroot)doc/manpages.xsl $<
build_doc_html: $(DOCS_HTML)
build_doc_man: $(DOCS_MAN3)
build_doc: $(DOCS)
#
# Include generated dependency files.
#
ifdef CC_MM
-include $(C_SYM_OBJS:%.$(O)=%.d)
-include $(C_OBJS:%.$(O)=%.d)
-include $(CPP_OBJS:%.$(O)=%.d)
-include $(C_PIC_OBJS:%.$(O)=%.d)
-include $(CPP_PIC_OBJS:%.$(O)=%.d)
-include $(C_JET_SYM_OBJS:%.$(O)=%.d)
-include $(C_JET_OBJS:%.$(O)=%.d)
-include $(C_TESTLIB_OBJS:%.$(O)=%.d)
-include $(TESTS_OBJS:%.$(O)=%.d)
-include $(TESTS_CPP_OBJS:%.$(O)=%.d)
endif
$(C_SYM_OBJS): $(objroot)src/%.sym.$(O): $(srcroot)src/%.c
$(C_SYM_OBJS): CPPFLAGS += -DJEMALLOC_NO_PRIVATE_NAMESPACE
$(C_SYMS): $(objroot)src/%.sym: $(objroot)src/%.sym.$(O)
$(C_OBJS): $(objroot)src/%.$(O): $(srcroot)src/%.c
$(CPP_OBJS): $(objroot)src/%.$(O): $(srcroot)src/%.cpp
$(C_PIC_OBJS): $(objroot)src/%.pic.$(O): $(srcroot)src/%.c
$(C_PIC_OBJS): CFLAGS += $(PIC_CFLAGS)
$(CPP_PIC_OBJS): $(objroot)src/%.pic.$(O): $(srcroot)src/%.cpp
$(CPP_PIC_OBJS): CXXFLAGS += $(PIC_CFLAGS)
$(C_JET_SYM_OBJS): $(objroot)src/%.jet.sym.$(O): $(srcroot)src/%.c
$(C_JET_SYM_OBJS): CPPFLAGS += -DJEMALLOC_JET -DJEMALLOC_NO_PRIVATE_NAMESPACE
$(C_JET_SYMS): $(objroot)src/%.jet.sym: $(objroot)src/%.jet.sym.$(O)
$(C_JET_OBJS): $(objroot)src/%.jet.$(O): $(srcroot)src/%.c
$(C_JET_OBJS): CPPFLAGS += -DJEMALLOC_JET
$(C_TESTLIB_UNIT_OBJS): $(objroot)test/src/%.unit.$(O): $(srcroot)test/src/%.c
$(C_TESTLIB_UNIT_OBJS): CPPFLAGS += -DJEMALLOC_UNIT_TEST
$(C_TESTLIB_INTEGRATION_OBJS): $(objroot)test/src/%.integration.$(O): $(srcroot)test/src/%.c
$(C_TESTLIB_INTEGRATION_OBJS): CPPFLAGS += -DJEMALLOC_INTEGRATION_TEST
$(C_UTIL_INTEGRATION_OBJS): $(objroot)src/%.integration.$(O): $(srcroot)src/%.c
$(C_TESTLIB_STRESS_OBJS): $(objroot)test/src/%.stress.$(O): $(srcroot)test/src/%.c
$(C_TESTLIB_STRESS_OBJS): CPPFLAGS += -DJEMALLOC_STRESS_TEST -DJEMALLOC_STRESS_TESTLIB
$(C_TESTLIB_OBJS): CPPFLAGS += -I$(srcroot)test/include -I$(objroot)test/include
$(TESTS_UNIT_OBJS): CPPFLAGS += -DJEMALLOC_UNIT_TEST
$(TESTS_INTEGRATION_OBJS): CPPFLAGS += -DJEMALLOC_INTEGRATION_TEST
$(TESTS_INTEGRATION_CPP_OBJS): CPPFLAGS += -DJEMALLOC_INTEGRATION_CPP_TEST
$(TESTS_STRESS_OBJS): CPPFLAGS += -DJEMALLOC_STRESS_TEST
$(TESTS_OBJS): $(objroot)test/%.$(O): $(srcroot)test/%.c
$(TESTS_CPP_OBJS): $(objroot)test/%.$(O): $(srcroot)test/%.cpp
$(TESTS_OBJS): CPPFLAGS += -I$(srcroot)test/include -I$(objroot)test/include
$(TESTS_CPP_OBJS): CPPFLAGS += -I$(srcroot)test/include -I$(objroot)test/include
ifneq ($(IMPORTLIB),$(SO))
$(CPP_OBJS) $(C_SYM_OBJS) $(C_OBJS) $(C_JET_SYM_OBJS) $(C_JET_OBJS): CPPFLAGS += -DDLLEXPORT
endif
# Dependencies.
ifndef CC_MM
HEADER_DIRS = $(srcroot)include/jemalloc/internal \
$(objroot)include/jemalloc $(objroot)include/jemalloc/internal
HEADERS = $(filter-out $(PRIVATE_NAMESPACE_HDRS),$(wildcard $(foreach dir,$(HEADER_DIRS),$(dir)/*.h)))
$(C_SYM_OBJS) $(C_OBJS) $(CPP_OBJS) $(C_PIC_OBJS) $(CPP_PIC_OBJS) $(C_JET_SYM_OBJS) $(C_JET_OBJS) $(C_TESTLIB_OBJS) $(TESTS_OBJS) $(TESTS_CPP_OBJS): $(HEADERS)
$(TESTS_OBJS) $(TESTS_CPP_OBJS): $(objroot)test/include/test/jemalloc_test.h
endif
$(C_OBJS) $(CPP_OBJS) $(C_PIC_OBJS) $(CPP_PIC_OBJS) $(C_TESTLIB_INTEGRATION_OBJS) $(C_UTIL_INTEGRATION_OBJS) $(TESTS_INTEGRATION_OBJS) $(TESTS_INTEGRATION_CPP_OBJS): $(objroot)include/jemalloc/internal/private_namespace.h
$(C_JET_OBJS) $(C_TESTLIB_UNIT_OBJS) $(C_TESTLIB_STRESS_OBJS) $(TESTS_UNIT_OBJS) $(TESTS_STRESS_OBJS): $(objroot)include/jemalloc/internal/private_namespace_jet.h
$(C_SYM_OBJS) $(C_OBJS) $(C_PIC_OBJS) $(C_JET_SYM_OBJS) $(C_JET_OBJS) $(C_TESTLIB_OBJS) $(TESTS_OBJS): %.$(O):
@mkdir -p $(@D)
$(CC) $(CFLAGS) -c $(CPPFLAGS) $(CTARGET) $<
ifdef CC_MM
@$(CC) -MM $(CPPFLAGS) -MT $@ -o $(@:%.$(O)=%.d) $<
endif
$(C_SYMS): %.sym:
@mkdir -p $(@D)
$(DUMP_SYMS) $< | $(AWK) -f $(objroot)include/jemalloc/internal/private_symbols.awk > $@
$(C_JET_SYMS): %.sym:
@mkdir -p $(@D)
$(DUMP_SYMS) $< | $(AWK) -f $(objroot)include/jemalloc/internal/private_symbols_jet.awk > $@
$(objroot)include/jemalloc/internal/private_namespace.gen.h: $(C_SYMS)
$(SHELL) $(srcroot)include/jemalloc/internal/private_namespace.sh $^ > $@
$(objroot)include/jemalloc/internal/private_namespace_jet.gen.h: $(C_JET_SYMS)
$(SHELL) $(srcroot)include/jemalloc/internal/private_namespace.sh $^ > $@
%.h: %.gen.h
@if ! `cmp -s $< $@` ; then echo "cp $< $<"; cp $< $@ ; fi
$(CPP_OBJS) $(CPP_PIC_OBJS) $(TESTS_CPP_OBJS): %.$(O):
@mkdir -p $(@D)
$(CXX) $(CXXFLAGS) -c $(CPPFLAGS) $(CTARGET) $<
ifdef CC_MM
@$(CXX) -MM $(CPPFLAGS) -MT $@ -o $(@:%.$(O)=%.d) $<
endif
ifneq ($(SOREV),$(SO))
%.$(SO) : %.$(SOREV)
@mkdir -p $(@D)
ln -sf $(<F) $@
endif
$(objroot)lib/$(LIBJEMALLOC).$(SOREV) : $(if $(PIC_CFLAGS),$(C_PIC_OBJS),$(C_OBJS)) $(if $(PIC_CFLAGS),$(CPP_PIC_OBJS),$(CPP_OBJS))
@mkdir -p $(@D)
$(CC) $(DSO_LDFLAGS) $(call RPATH,$(RPATH_EXTRA)) $(LDTARGET) $+ $(LDFLAGS) $(LIBS) $(EXTRA_LDFLAGS)
$(objroot)lib/$(LIBJEMALLOC)_pic.$(A) : $(C_PIC_OBJS) $(CPP_PIC_OBJS)
$(objroot)lib/$(LIBJEMALLOC).$(A) : $(C_OBJS) $(CPP_OBJS)
$(objroot)lib/$(LIBJEMALLOC)_s.$(A) : $(C_OBJS) $(CPP_OBJS)
$(STATIC_LIBS):
@mkdir -p $(@D)
$(AR) $(ARFLAGS)@AROUT@ $+
$(objroot)test/unit/%$(EXE): $(objroot)test/unit/%.$(O) $(C_JET_OBJS) $(C_TESTLIB_UNIT_OBJS)
@mkdir -p $(@D)
$(CC) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(LDFLAGS) $(filter-out -lm,$(LIBS)) $(LM) $(EXTRA_LDFLAGS)
$(objroot)test/integration/%$(EXE): $(objroot)test/integration/%.$(O) $(C_TESTLIB_INTEGRATION_OBJS) $(C_UTIL_INTEGRATION_OBJS) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB)
@mkdir -p $(@D)
$(CC) $(TEST_LD_MODE) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(LJEMALLOC) $(LDFLAGS) $(filter-out -lm,$(filter -lrt -lpthread -lstdc++,$(LIBS))) $(LM) $(EXTRA_LDFLAGS)
$(objroot)test/integration/cpp/%$(EXE): $(objroot)test/integration/cpp/%.$(O) $(C_TESTLIB_INTEGRATION_OBJS) $(C_UTIL_INTEGRATION_OBJS) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB)
@mkdir -p $(@D)
$(CXX) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB) $(LDFLAGS) $(filter-out -lm,$(LIBS)) -lm $(EXTRA_LDFLAGS)
$(objroot)test/stress/%$(EXE): $(objroot)test/stress/%.$(O) $(C_JET_OBJS) $(C_TESTLIB_STRESS_OBJS) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB)
@mkdir -p $(@D)
$(CC) $(TEST_LD_MODE) $(LDTARGET) $(filter %.$(O),$^) $(call RPATH,$(objroot)lib) $(objroot)lib/$(LIBJEMALLOC).$(IMPORTLIB) $(LDFLAGS) $(filter-out -lm,$(LIBS)) $(LM) $(EXTRA_LDFLAGS)
build_lib_shared: $(DSOS)
build_lib_static: $(STATIC_LIBS)
build_lib: build_lib_shared build_lib_static
install_bin:
$(INSTALL) -d $(BINDIR)
@for b in $(BINS); do \
echo "$(INSTALL) -m 755 $$b $(BINDIR)"; \
$(INSTALL) -m 755 $$b $(BINDIR); \
done
install_include:
$(INSTALL) -d $(INCLUDEDIR)/jemalloc
@for h in $(C_HDRS); do \
echo "$(INSTALL) -m 644 $$h $(INCLUDEDIR)/jemalloc"; \
$(INSTALL) -m 644 $$h $(INCLUDEDIR)/jemalloc; \
done
install_lib_shared: $(DSOS)
$(INSTALL) -d $(LIBDIR)
$(INSTALL) -m 755 $(objroot)lib/$(LIBJEMALLOC).$(SOREV) $(LIBDIR)
ifneq ($(SOREV),$(SO))
ln -sf $(LIBJEMALLOC).$(SOREV) $(LIBDIR)/$(LIBJEMALLOC).$(SO)
endif
install_lib_static: $(STATIC_LIBS)
$(INSTALL) -d $(LIBDIR)
@for l in $(STATIC_LIBS); do \
echo "$(INSTALL) -m 755 $$l $(LIBDIR)"; \
$(INSTALL) -m 755 $$l $(LIBDIR); \
done
install_lib_pc: $(PC)
$(INSTALL) -d $(LIBDIR)/pkgconfig
@for l in $(PC); do \
echo "$(INSTALL) -m 644 $$l $(LIBDIR)/pkgconfig"; \
$(INSTALL) -m 644 $$l $(LIBDIR)/pkgconfig; \
done
install_lib: install_lib_shared install_lib_static install_lib_pc
install_doc_html:
$(INSTALL) -d $(DATADIR)/doc/jemalloc$(install_suffix)
@for d in $(DOCS_HTML); do \
echo "$(INSTALL) -m 644 $$d $(DATADIR)/doc/jemalloc$(install_suffix)"; \
$(INSTALL) -m 644 $$d $(DATADIR)/doc/jemalloc$(install_suffix); \
done
install_doc_man:
$(INSTALL) -d $(MANDIR)/man3
@for d in $(DOCS_MAN3); do \
echo "$(INSTALL) -m 644 $$d $(MANDIR)/man3"; \
$(INSTALL) -m 644 $$d $(MANDIR)/man3; \
done
install_doc: install_doc_html install_doc_man
install: install_bin install_include install_lib install_doc
tests_unit: $(TESTS_UNIT:$(srcroot)%.c=$(objroot)%$(EXE))
tests_integration: $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%$(EXE)) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%$(EXE))
tests_stress: $(TESTS_STRESS:$(srcroot)%.c=$(objroot)%$(EXE))
tests: tests_unit tests_integration tests_stress
check_unit_dir:
@mkdir -p $(objroot)test/unit
check_integration_dir:
@mkdir -p $(objroot)test/integration
stress_dir:
@mkdir -p $(objroot)test/stress
check_dir: check_unit_dir check_integration_dir
check_unit: tests_unit check_unit_dir
$(SHELL) $(objroot)test/test.sh $(TESTS_UNIT:$(srcroot)%.c=$(objroot)%)
check_integration_prof: tests_integration check_integration_dir
ifeq ($(enable_prof), 1)
$(MALLOC_CONF)="prof:true" $(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%)
$(MALLOC_CONF)="prof:true,prof_active:false" $(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%)
endif
check_integration_decay: tests_integration check_integration_dir
$(MALLOC_CONF)="dirty_decay_ms:-1,muzzy_decay_ms:-1" $(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%)
$(MALLOC_CONF)="dirty_decay_ms:0,muzzy_decay_ms:0" $(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%)
check_integration: tests_integration check_integration_dir
$(SHELL) $(objroot)test/test.sh $(TESTS_INTEGRATION:$(srcroot)%.c=$(objroot)%) $(TESTS_INTEGRATION_CPP:$(srcroot)%.cpp=$(objroot)%)
stress: tests_stress stress_dir
$(SHELL) $(objroot)test/test.sh $(TESTS_STRESS:$(srcroot)%.c=$(objroot)%)
check: check_unit check_integration check_integration_decay check_integration_prof
clean:
rm -f $(PRIVATE_NAMESPACE_HDRS)
rm -f $(PRIVATE_NAMESPACE_GEN_HDRS)
rm -f $(C_SYM_OBJS)
rm -f $(C_SYMS)
rm -f $(C_OBJS)
rm -f $(CPP_OBJS)
rm -f $(C_PIC_OBJS)
rm -f $(CPP_PIC_OBJS)
rm -f $(C_JET_SYM_OBJS)
rm -f $(C_JET_SYMS)
rm -f $(C_JET_OBJS)
rm -f $(C_TESTLIB_OBJS)
rm -f $(C_SYM_OBJS:%.$(O)=%.d)
rm -f $(C_OBJS:%.$(O)=%.d)
rm -f $(CPP_OBJS:%.$(O)=%.d)
rm -f $(C_PIC_OBJS:%.$(O)=%.d)
rm -f $(CPP_PIC_OBJS:%.$(O)=%.d)
rm -f $(C_JET_SYM_OBJS:%.$(O)=%.d)
rm -f $(C_JET_OBJS:%.$(O)=%.d)
rm -f $(C_TESTLIB_OBJS:%.$(O)=%.d)
rm -f $(TESTS_OBJS:%.$(O)=%$(EXE))
rm -f $(TESTS_OBJS)
rm -f $(TESTS_OBJS:%.$(O)=%.d)
rm -f $(TESTS_OBJS:%.$(O)=%.out)
rm -f $(TESTS_CPP_OBJS:%.$(O)=%$(EXE))
rm -f $(TESTS_CPP_OBJS)
rm -f $(TESTS_CPP_OBJS:%.$(O)=%.d)
rm -f $(TESTS_CPP_OBJS:%.$(O)=%.out)
rm -f $(DSOS) $(STATIC_LIBS)
distclean: clean
rm -f $(objroot)bin/jemalloc-config
rm -f $(objroot)bin/jemalloc.sh
rm -f $(objroot)bin/jeprof
rm -f $(objroot)config.log
rm -f $(objroot)config.status
rm -f $(objroot)config.stamp
rm -f $(cfghdrs_out)
rm -f $(cfgoutputs_out)
relclean: distclean
rm -f $(objroot)configure
rm -f $(objroot)VERSION
rm -f $(DOCS_HTML)
rm -f $(DOCS_MAN3)
#===============================================================================
# Re-configuration rules.
ifeq ($(enable_autogen), 1)
$(srcroot)configure : $(srcroot)configure.ac
cd ./$(srcroot) && $(AUTOCONF)
$(objroot)config.status : $(srcroot)configure
./$(objroot)config.status --recheck
$(srcroot)config.stamp.in : $(srcroot)configure.ac
echo stamp > $(srcroot)config.stamp.in
$(objroot)config.stamp : $(cfgoutputs_in) $(cfghdrs_in) $(srcroot)configure
./$(objroot)config.status
@touch $@
# There must be some action in order for make to re-read Makefile when it is
# out of date.
$(cfgoutputs_out) $(cfghdrs_out) : $(objroot)config.stamp
@true
endif
+20
View File
@@ -0,0 +1,20 @@
jemalloc is a general purpose malloc(3) implementation that emphasizes
fragmentation avoidance and scalable concurrency support. jemalloc first came
into use as the FreeBSD libc allocator in 2005, and since then it has found its
way into numerous applications that rely on its predictable behavior. In 2010
jemalloc development efforts broadened to include developer support features
such as heap profiling and extensive monitoring/tuning hooks. Modern jemalloc
releases continue to be integrated back into FreeBSD, and therefore versatility
remains critical. Ongoing development efforts trend toward making jemalloc
among the best allocators for a broad range of demanding applications, and
eliminating/mitigating weaknesses that have practical repercussions for real
world applications.
The COPYING file contains copyright and licensing information.
The INSTALL file contains information on how to configure, build, and install
jemalloc.
The ChangeLog file contains a brief summary of changes for each release.
URL: http://jemalloc.net/
+17
View File
@@ -0,0 +1,17 @@
#!/bin/sh
for i in autoconf; do
echo "$i"
$i
if [ $? -ne 0 ]; then
echo "Error $? in $i"
exit 1
fi
done
echo "./configure --enable-autogen $@"
./configure --enable-autogen $@
if [ $? -ne 0 ]; then
echo "Error $? in ./configure"
exit 1
fi
+83
View File
@@ -0,0 +1,83 @@
#!/bin/sh
usage() {
cat <<EOF
Usage:
@BINDIR@/jemalloc-config <option>
Options:
--help | -h : Print usage.
--version : Print jemalloc version.
--revision : Print shared library revision number.
--config : Print configure options used to build jemalloc.
--prefix : Print installation directory prefix.
--bindir : Print binary installation directory.
--datadir : Print data installation directory.
--includedir : Print include installation directory.
--libdir : Print library installation directory.
--mandir : Print manual page installation directory.
--cc : Print compiler used to build jemalloc.
--cflags : Print compiler flags used to build jemalloc.
--cppflags : Print preprocessor flags used to build jemalloc.
--cxxflags : Print C++ compiler flags used to build jemalloc.
--ldflags : Print library flags used to build jemalloc.
--libs : Print libraries jemalloc was linked against.
EOF
}
prefix="@prefix@"
exec_prefix="@exec_prefix@"
case "$1" in
--help | -h)
usage
exit 0
;;
--version)
echo "@jemalloc_version@"
;;
--revision)
echo "@rev@"
;;
--config)
echo "@CONFIG@"
;;
--prefix)
echo "@PREFIX@"
;;
--bindir)
echo "@BINDIR@"
;;
--datadir)
echo "@DATADIR@"
;;
--includedir)
echo "@INCLUDEDIR@"
;;
--libdir)
echo "@LIBDIR@"
;;
--mandir)
echo "@MANDIR@"
;;
--cc)
echo "@CC@"
;;
--cflags)
echo "@CFLAGS@"
;;
--cppflags)
echo "@CPPFLAGS@"
;;
--cxxflags)
echo "@CXXFLAGS@"
;;
--ldflags)
echo "@LDFLAGS@ @EXTRA_LDFLAGS@"
;;
--libs)
echo "@LIBS@"
;;
*)
usage
exit 1
esac
+9
View File
@@ -0,0 +1,9 @@
#!/bin/sh
prefix=@prefix@
exec_prefix=@exec_prefix@
libdir=@libdir@
@LD_PRELOAD_VAR@=${libdir}/libjemalloc.@SOREV@
export @LD_PRELOAD_VAR@
exec "$@"
File diff suppressed because it is too large. Load diff
+1462
View File
File diff suppressed because it is too large. Load diff
+1825
View File
File diff suppressed because it is too large. Load diff
Loaded 100 of 643 files, more files were not shown because too many files have changed in this diff. Show more